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Abstract 

The  U.S.  Air  Force  has  recognized  that  organizations  across  Air  Force  (AF) 
installations  often  require  geospatial  information  resources  or  maps  to  accomplish 
mission  essential  tasks.  To  provide  an  AF-wide  approach  to  addressing  this  need,  in 
2001,  the  Headquarters  Air  Force  Geo  Integration  Office  formed  the  USAF  GeoBase 
program.  GeoBase  is  the  name  given  to  the  Air  Force’s  GIS  program.  Although  the  AF 
has  specific  needs  and  requirements  for  GIS,  the  foundation  for  the  AF  GeoBase  program 
remains  in  the  use  of  private  sector  GIS  technology.  As  GeoBase  has  been  implemented 
across  the  AF,  there  have  been  no  mandated  product  standards  for  GIS  applications. 

This  thesis  focuses  on  two  different  GIS  applications  being  used  across  the  Civil 
Engineer  (CE)  community  for  the  management  of  airfield  obstructions.  These  two 
applications  are  the  Airfield  Obstruction  Management  System  (AOMS)  and  the  Airfield 
Obstruction  Tracking,  Analysis,  and  Management  System  (AIROBS).  In  addition  to  the 
development  of  a  unique  methodological  approach  for  accomplishing  analyses,  this 
research  presents  how  each  application  rates  in  usability  of  accomplishing  tasks  and  the 
level  of  satisfaction  as  determined  by  the  end-user.  Overall,  the  results  revealed  that 
AOMS  was  identified  as  having  fewer  problems  in  usability  and  rated  slightly  higher  in 
End-User  Computing  Satisfaction.  The  methodology  is  also  offered  as  a  way  to  compare 
other  GIS  applications  where  there  is  an  intent  to  determine  the  best  application  for  a 
specific  mission  purpose. 
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A  COMPARATIVE  USABILITY  AND  END-USER  SATISFACTION  ANALYSIS 


OF  TWO  GEOGRAPHIC  INFORMATION  SYSTEM  (GIS)  APPLICATIONS 


I.  Introduction 


Background 

A  Geographic  Information  System  (GIS)  is  a  specialized  class  of  database 
management  system  that  allows  users  to  analyze,  relate,  and  display  spatial  attributes  of 
data  in  addition  to  conventional  relational  data  (West,  2000).  The  Environmental 
Systems  Research  Institute  (ESRI)  defines  GIS  as  “an  organized  collection  of  computer 
hardware,  software,  geographical  data  and  personnel  designed  to  efficiently  capture, 
store,  update,  manipulate,  analyze,  and  display  all  forms  of  geographically  referenced 
materials”  (Heikkila,  1998).  The  purpose  of  GIS  software  applications  is  to  perfonn  the 
same  kinds  of  spatial  analyses  once  done  by  hand,  but  to  do  them  with  much  greater 
speed  and  accuracy  (Gilbrook,  1999).  GIS  can  display  maps  on  a  computer  screen  and 
can  also  provide  detailed  information  about  map  features,  including  roads,  buildings, 
streams,  etc.  GIS  are  being  used  to  store  feature  attributes  and  analyze  the  “spatial” 
relationships  which  can  help  manage  forests,  utilities,  and  petroleum  exploration  (Korte, 
2001).  As  such,  public  corporations;  federal,  state,  and  local  governments;  and  even  the 
military  is  using  GIS  technology. 

The  U.S.  Air  Force  has  also  recognized  that  organizations  across  AF  installations 
often  require  geospatial  information  resources  or  maps  to  accomplish  mission  essential 
tasks  such  as  land  use  planning,  identifying  disaster  response  cordon/stand-off  distances 
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and  emergency  response  routes,  and  locating  underground  utilities  just  to  name  a  few.  To 
provide  an  AF-wide  approach  to  addressing  this  need,  in  2001,  the  Headquarters  Air 
Force  Geo  Integration  Office  (HAF  GIO)  formed  the  USAF  GeoBase  program.  GeoBase 
is  the  name  given  to  the  Air  Force’s  GIS  program.  In  2002,  wing-level  CE  organizations 
were  identified  as  the  central  point  of  contact  for  all  base-level  mapping  requirements  and 
were  granted  responsibility  for  the  installation  GeoBase  program  by  the  Office  of  the 
Civil  Engineer,  Installation  and  Logistics,  Headquarters,  United  States  Air  Force  (ILE). 
Although  the  AF  has  specific  needs  and  requirements  for  GIS,  the  foundation  for  the  AF 
GeoBase  program  remains  in  the  use  of  private  sector  GIS  technology.  The  specific 
mission  of  the  USAF  GeoBase  program  is  to  “attain,  maintain,  and  sustain  one  geospatial 
infostructure  to  address  Air  Force  installation  requirements,”  with  a  vision  toward  “one 
installation,  one  map”  (Zettler,  2002).  The  GeoBase  program  is  built  around  three 
conceptual  views:  Strategic,  Expeditionary  (also  known  as  GeoReach),  and  Garrison. 
Strategic  GeoBase  is  a  generalized  view  of  AF  installations  served  to  agencies  beyond 
the  Major  Commands  (MAJCOM)  via  a  central  repository  of  digital  GeoBase  data  at  the 
Pentagon.  Expeditionary  GeoBase  is  a  forward  deployed  version  of  Garrison  GeoBase 
with  capabilities  affording  commanders  and  ainnen  enhanced  situational  awareness  of 
expeditionary  bases.  Garrison  GeoBase  consolidates  an  installation’s  mapping  needs  and 
provides  a  common  digital  map  throughout  the  installation  network.  In  addition, 

Garrison  GeoBase  allows  a  command  and  control  capability  that  provides  commanders 
and  first  responders  increased  ability  to  respond  to  emergency  situations  by  visualizing 
incident  location  and  status  in  relation  to  base  assets  and  resources  using  a  common  map. 
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The  installation  geospatial  data  is  organized  into  layers  that  represent  buildings,  roads, 
airfield  surfaces,  etc.  and  are  then  synthesized  into  the  Common  Installation  Picture 
(CIP).  The  CIP  is  the  high-fidelity  base  map  that  can  be  viewed  for  reference  by  all 
functional  communities  at  a  given  installation  (HAF  GIO,  2003a,  p.  5). 

As  GeoBase  has  been  implemented  across  the  AF,  there  have  been  no  mandated 
product  standards  for  GIS  applications.  No  single  GeoBase  application  for  a  specific 
purpose,  such  as  managing  airfield  obstructions,  exists  for  use  across  the  CE  community 
(HAF  GIO,  2003a,  p.  17).  As  such,  MAJCOMs  and  installations  are  using  different 
vendors  for  various  GIS  applications.  This  has  led  to  multiple  software  applications 
being  developed  and  purchased  for  similar  uses.  In  turn,  this  has  caused  general 
difficulty  in  the  CE  leadership’s  visibility  and  management  of  the  many  applications  in 
use  across  the  MAJCOMs  and  installations.  This  has  caused  MAJCOMs  and 
installations  to  purchase  and/or  have  applications  developed  for  specific  purposes, 
whereas  a  suitable  application  may  already  exist.  At  this  point,  one  must  wonder  if  one 
application  is  better  than  another  and,  if  so,  should  the  better  application  be  mandated  for 
that  purpose.  This  research  focuses  on  analyzing  two  such  GeoBase  applications  that  are 
used  by  CE  organizations  across  the  AF. 

Problem  and  Purpose  Statement 

Currently  there  are  two  different  GIS  applications  being  used  across  the  CE 
community  for  the  management  of  airfield  obstructions.  These  two  applications  are  the 
Airfield  Obstruction  Management  System  (AOMS)  and  the  Airfield  Obstruction 
Tracking,  Analysis,  and  Management  System  (AIROBS).  The  two  applications  are  being 
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implemented  and  funded  by  MAJCOMs  and  installations  without  any  funding  oversight 
or  management  by  the  Headquarters  Air  Force  Geo  Integration  Office  (HAF  GIO).  Since 
the  GeoBase  program  started  in  2002,  the  need  to  understand  what  GIS  applications  are 
in  use  across  the  AF  and  the  management  of  those  applications  by  the  HAF  GIO  has 
become  increasingly  important  to  ensure  that  application  purchasing  and  development 
can  be  leveraged  across  the  AF.  As  such,  this  research  project  will  attempt  to  complete  a 
comparative  analysis  of  AOMS  and  AIROBS  to  be  used  as  a  foundation  in  any  future 
discussions  regarding  the  implementation  of  only  one  application  across  the  AF.  The 
analysis  will  address  how  each  application  rates  in  usability  of  accomplishing  tasks  and 
the  level  of  satisfaction  as  detennined  by  the  end-user.  To  carry  out  this  research,  the 
following  research  questions  will  be  addressed. 

Research  Questions 

1 .  How  do  AOMS  and  AIROBS  compare  in  software  usability,  where  usability 
criteria  measures  error  abatement,  responsiveness,  descriptiveness, 
consistency,  and  simplicity? 

2.  How  do  AOMS  and  AIROBS  compare  in  end-user  computing  satisfaction 
(EUCS),  as  determined  by  the  measures  of  content,  format,  and  ease  of  use 
provided  by  each  application? 

Methodology 

To  address  the  research  questions,  the  method  of  analysis  will  consist  of  an 
evaluation  questionnaire  focusing  on  software  usability  and  end-user  computing 
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satisfaction.  Task  scenarios  will  be  used  to  provide  the  evaluators  some  interactive 
situations  with  the  applications  prior  to  answering  the  questionnaire.  The  task  scenarios 
were  developed  based  on  the  application’s  purpose  of  tracking,  analyzing,  and  managing 
airfield  obstructions  and  input  from  AF  community  planners.  The  evaluation 
questionnaire  is  a  combination  of  software  usability  and  EUCS  criteria.  The  software 
usability  evaluation  is  a  non-statistical  method  aimed  at  identifying  problem  areas  within 
the  applications  by  measuring  software  functions  which  are  not  always  present  when 
performing  the  task  scenarios.  The  EUCS  items  are  closed-ended  responses  that  describe 
the  level  of  satisfaction  provided  to  the  evaluators  by  the  application.  These  responses 
will  be  analyzed  using  descriptive  statistics  (Doll  &  Torkzadeh,  1988,  p.  263).  The 
population  for  the  evaluations  will  include  CE  officers  and  CE  civilians  who  are  either 
AFIT  graduate  students  or  instructors  assigned  to  the  AFIT  Civil  Engineer  and  Services 
School. 

Benefits/Implications  of  Research 

This  research  will  be  the  first  known  attempt  in  the  GeoBase  arena  to  compare 
two  applications  being  used  for  the  same  purpose.  The  evaluations  will  assess 
application  usability  and  end-user  satisfaction  which  may  identify  which  application  is 
perceived  to  be  more  effective  in  task/requirement  accomplishment  by  respondents.  The 
results  of  this  study  may  assist  in  identifying  a  standard  application  for  airfield 
obstruction  management  that  could  be  implemented  AF  wide.  In  addition,  this  research 
methodology  may  lend  itself  to  be  used  with  other  application  comparison  analyses. 
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Thesis  Overview 


Chapter  one  provides  a  background  for  the  thesis  research.  In  chapter  two,  a 
literature  review  of  GIS  and  how  GIS  are  being  used  in  the  AF  will  be  provided.  Chapter 
three  will  detail  the  research  methodology  used,  along  with  how  the  data  was  collected 
and  analyzed.  The  fourth  chapter  will  present  the  results  and  analysis  of  the  findings 
based  on  the  research  questions.  In  the  last  chapter,  a  discussion  of  the  results  which  may 
be  used  to  guide  decision  making  will  be  presented.  In  addition,  limitations  of  this 
research  and  suggestions  for  future  research  will  be  given. 
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II.  Literature  Review 


An  analysis  of  airfield  obstruction  management  systems  must  begin  by  exploring 
the  history  of  Information  Systems  (IS)  and  Geographic  Infonnation  Systems  (GIS). 

This  literature  review  will  take  a  historical  look  at  IS  and  how  these  systems  have 
progressed  over  the  last  40  years.  In  addition,  the  evolution  of  GIS  and  the  similarities 
GIS  holds  with  traditional  IS  will  be  presented.  Then,  a  discussion  of  GIS  in  the  AF,  or 
GeoBase,  will  be  presented.  The  next  area  of  the  chapter  will  describe  airfield 
obstructions  and  airfield  surfaces.  Finally,  airfield  obstruction  management  systems  will 
be  discussed.  This  discussion  will  include  why  and  how  airfield  management  has 
become  so  important.  Furthennore,  the  two  airfield  obstruction  management  systems  in 
use  by  the  Air  Force  will  be  profiled. 

Information  Systems 

“An  information  system  or  IS  is  a  system  designed  to  collect,  store,  manipulate, 
and  analyze  information  and  then  use  the  information  for  the  purpose  that  it  was 
collected”  (Pittman,  1990,  p.  4).  Laudon  and  Laudon  define  an  infonnation  system  “as  a 
set  of  interrelated  components  that  collect  (or  retrieve),  process,  store,  and  distribute 
information  to  support  decision  making  and  control  in  organizations”  (Laudon,  1998,  p. 
7).  Worboys  and  Duckham  further  define  an  infonnation  system  as  “an  association  of 
people,  machines,  data,  and  procedures  working  together  to  collect,  manage,  and 
distribute  information  of  importance  to  individuals  or  organizations”  (Worboys  & 
Duckham,  2004,  p.  1).  The  relationship  between  the  computer  and  infonnation  was  the 
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theme  of  Edward  Berkeley’s  1949  book,  Giant  Brains,  or  Machines  That  Think.  This 
was  the  first  book  to  make  the  connection  between  computers  and  their  potential  use  in 
business  (Haigh,  2001,  p.  32).  Early  information  systems,  from  the  1950s  through  the 
mid-1960s,  were  mostly  oriented  toward  data  processing  and  hardware/software 
technologies  (Lee,  1988,  p.  17).  It  was  during  the  late  1950s  and  early  1960s  that  the 
concept  of  the  “totally  integrated  management  information  system”  came  into  being.  The 
idea  was  to  have  an  integrated  computer  system  designed  to  encompass  all  administrative 
and  managerial  activities  (Haigh,  2001,  p.  15).  During  the  data  processing  era,  most 
applications  were  used  at  the  operations  level  such  as  in  the  finance  departments.  As  the 
technology  changed  the  applications  also  began  to  move  from  mainframes,  to 
minicomputers,  and  on  to  personal  computers.  These  changes  brought  about  the 
decentralization  of  the  information  systems  organization  (Lee,  1988,  p.  18).  Infonnation 
systems  now  play  a  larger  role  in  organizational  life.  Today,  infonnation  systems  are 
rooted  in  organizational  strategy  and  daily  operations  (Laudon,  1998,  p.  29). 

GIS-Another  Type  of  IS 

“A  GIS  is  a  special  type  of  information  system  concerned  with  geographically 
referenced  data”  (Worboys  &  Duckham,  2004,  p.2).  A  GIS  has  two  distinguishing 
characteristics  that  make  it  different  from  a  standard  information  system.  First,  the  data 
in  a  GIS  are  “spatially  referenced,  usually  with  x-y  or  latitude-longitude  coordinates.” 
Second,  a  GIS  will  normally  have  mapping  capabilities  associated  with  them  (Pittman, 
1990,  p.  4).  As  with  a  standard  IS,  the  central  part  of  any  GIS  is  the  database.  Because 
the  data  is  geographically  referenced,  the  GIS  data  sets  are  usually  larger  and  more 
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complex  than  other  IS’s  (Worboys  &  Duckham,  2004,  p.3).  The  database  contains  map 
layers  (Figure  1)  that  represent  geographic  features,  which  are  referenced  to  a  standard 
coordinate  system  such  as  the  Universal  Transverse  Mercator  (UTC)  or  State  Plane 
Coordinate  system  (Fung  &  Remsen,  1997,  p.  18).  With  this  type  of  mapping,  any  one 


Street  Network 
Housing  Zones 

Houses  Sold  (1980) 

Houses  Sold  (1990) 


Base  Map 


Figure  1.  A  GIS  database  may  be  conceptualized  as  a  stack  of  floating  map  layers 
registered  to  a  common  map  base.  (Adapted  from  Fung  &  Remsen,  1997) 

of  the  layers  can  be  accessed  independently  or  in  combination  with  other  layers  (Fung  & 
Remsen,  1997,  p.  18).  A  primary  function  of  GIS  tools  is  to  perform  spatial  analyses 
with  more  accuracy  and  speed  than  was  previously  done  by  hand  (Gilbrook,  1999,  p.  34). 
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GIS  History  and  Evolution 

As  was  stated  in  Chapter  1 ,  the  Environmental  Systems  Research  Institute  (ESRI) 
defines  GIS  as  “an  organized  collection  of  computer  hardware,  software,  geographical 
data  and  personnel  designed  to  efficiently  capture,  store,  update,  manipulate,  analyze  and 
display  all  forms  of  geographically  referenced  materials”  (Heikkila,  1998).  This 
definition  includes  technology  and  people,  but  the  essence  for  GIS  evolution  lies  in 
computer  technology.  As  technology  has  advanced  and  expanded  across  wider  markets, 
more  applications  are  being  developed  to  handle  the  spatial  information.  Automated  GIS 
began  in  the  1960s  with  the  Canadian  forestry  service.  Dr.  Roger  Tomlinson  led  the 
development  of  the  first  industry-wide  automated  GIS,  known  as  the  Canadian 
Geographic  Information  System.  After  this  initiative,  the  term  “geographic  information 
system”  became  widespread  (Foresman,  1998,  p.  10).  GISs  continued  to  expand  as  North 
American  government  and  university  researchers  sought  to  develop  methods  to  represent 
the  earth’s  geography  using  a  computer  database,  display  it  on  a  computer  screen,  and 
print  it  on  paper  (Korte,  2001,  p.  6).  In  the  U.S.,  the  National  Environmental  Policy  Act 
(NEPA)  of  1970  continued  the  trend  toward  increased  land  use  management  and 
environmental  protection  that  led  to  many  mandated  programs  that  bolstered  GIS 
technological  development.  NEPA  is  recognized  as  having  the  most  influence  for 
advancing  the  use  of  GIS  in  the  federal  government  (Foresman,  1998,  p.  10).  Several 
corporations  were  founded  in  the  1970s  to  develop  and  sell  systems  for  computer 
mapping  and  analysis.  Initially,  only  the  largest  government  agencies  and  corporations 
could  afford  GIS.  However,  in  the  1980s,  the  GIS  market  continued  to  grow  because  of 
the  benefits  provided  to  organizations.  From  the  late  1980s  through  the  1990s,  personal 
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computers  and  the  World  Wide  Web  have  expanded  the  use  of  GIS  to  practically  anyone 
(Korte,  2001,  p.  6). 

GIS  in  the  Air  Force 

As  stated  previously,  the  Air  Force’s  GIS  program,  or  GeoBase,  started  in  2002 
by  direction  of  the  AF  Civil  Engineer.  The  guiding  documents  for  the  GeoBase  program 
are  the  USAF  Garrison  Mapping  Concept  of  Operations  (CONOPS)  Version  2.0  (  2003), 
the  USAF  GeoBase  Common  Installation  Picture  Data  Model  Standardization  Work  Plan 
Version  1.0  (2003),  and  the  USAF  GeoBase  Enterprise  Architecture  Version  1.0  (2003). 
The  vision  of  the  AF  GeoBase  program  is  “one  installation,  one  map  with  a  mission  to 
attain  and  sustain  a  breakthrough  capability  enabling  shared,  efficient  use  of  trusted, 
integrated,  and  georeferenced  information  delivering  situational  awareness  across 
installations”  (HAF  GIO,  2003a,  p.4). 

GeoBase  Standards 

In  October  1990,  the  Office  of  Management  and  Budget  established  the  Federal 
Geographic  Data  Committee  (FGDC)  to  further  the  development,  use,  and  sharing  of 
geographic  data  within  and  across  the  federal  agencies  and  departments  (Mangan,  1995, 
p.  99).  In  1992,  in  response  to  the  need  for  standardization  policy  across  Department  of 
Defense  (DOD)  installations,  the  DoD,  along  with  FGDC,  created  the  Computer-Aided 
Design  and  Drafting  (CADD)/GIS  Technology  Center  to  advance  GIS  technology  across 
the  DoD  (Korte,  2001,  p.  84).  The  Center  developed  and  now  annually  updates  the 
Spatial  Data  Standards  for  Facilities,  Infrastructure,  and  Environment  (SDSFIE)  ("Spatial 
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Data  Standard",  2005).  The  SDSFIE  provides  standardized  groups  and  names  for 
geographically  referenced  features  and  also  provides  an  attribute  table  containing  data 
about  the  geospatial  feature  (Korte,  2001,  p.  85).  The  SDSFIE  is  a  nonproprietary  GIS 
standard  designed  to  be  used  with  commercial  GIS  software.  As  a  result,  the  DoD  has 
adopted  the  SDSFIE  as  the  standard  for  GIS  implementation  ("Spatial  Data  Standard", 
2005).  In  2003,  the  USAF  GeoBase  Enterprise  Architecture  Version  1. 0  was  developed 
to  “guide  USAF  organizations  in  the  process  of  selecting  IT  standards  and  technologies 
to  deploy  and  exploit  GeoBase  capabilities.”  The  GeoBase  architecture  is  based  on  the 
Air  Force  Command,  Control,  Communications,  Computers,  and  Intelligence, 
Surveillance,  and  Reconnaissance  (C4ISR)  Architecture  Plan,  developed  by  the  Deputy 
Chief  of  Staff  for  Warfighting  Integration,  Headquarters  U.S.  Air  Force  (HAF  GIO, 
2003b,  p.  iii).  The  C4ISR  Architecture  Plan  has  been  replaced  by  the  Department  of 
Defense  Architecture  Framework  (DoDAF)  Version  1.0,  dated  August  2003.  These 
standards  have  been  applied  to  GeoBase  applications  such  as  airfield  obstruction 
management  systems. 

Airfield  Obstruction  Reduction  Initiative  Background 

On  July  24,  1998,  at  Misawa  Air  Base  (AB),  Japan,  an  F-16  pilot  aborted  takeoff 
and  the  F-16  crashed  off  the  end  of  the  runway.  The  pilot  ejected  when  the  aircraft  speed 
was  at  56  knots.  The  aircraft  went  off  the  runway  and  struck  various  structures  of  the 
approach  lighting  and  instrument  landing  system  (ILS)  localizer  antenna  systems.  The 
external  fuel  tanks  were  damaged  and  the  aircraft  caught  fire.  The  pilot  drifted  into  the 
fire  before  reaching  the  ground.  The  pilot  was  fatally  injured  from  the  burns  (Holliday, 
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2001,  p.  1).  From  August  -  October  1999,  SAF/IG  conducted  a  review  of  construction 
and  repair  projects  at  Misawa  AB.  In  response  to  the  Secretary  of  the  Air  Force  Inspector 
General  (SAF/IG)  Report  of  Review,  the  Air  Force  Chief  of  Staff  directed  that  corrective 
action  be  taken  to  reduce  airfield  obstructions.  Based  on  this  directive,  AF/ILE  instructed 
that  all  AF  bases  identify  airfield  obstructions  and  directed  a  plan  be  developed  to 
mitigate  airfield  obstructions  (Air  Force  Tiger  Team,  2000  p.  7).  Th q  Airfield 
Obstruction  Reduction  Initiative  (AORI)  Report,  dated  Nov  2000,  is  the  response  to  the 
SAF/IG  inspection.  HQ  AF/ILE  assigned  a  team  to  collect  and  analyze  the  data  provided 
by  the  bases.  The  primary  purpose  of  the  team  was  to  investigate  possible  problems  and 
make  recommendations  for  improvement  (Ates,  2001).  During  the  AF/ILE  tiger  team 
assessment,  the  team  recommended  that  AF/ILE  establish  “standards  (and  perhaps  a 
standardized  system)  for  collecting,  and  reporting,  AF-wide  airfield  obstruction  waiver 
data.”  At  the  time  of  this  assessment,  there  was  “no  standardized  format  for  collecting 
and  reporting  annual  waiver  data”  (Air  Force  Tiger  Team,  2000,  p.  18).  This  lack  of  a 
standard  reduced  the  effective  management  of  airfield  obstructions  that  required  a  waiver. 
The  waivers  can  be  pennanent  or  temporary  and  are  used  to  allow  deviations  from  the 
standards.  This  ineffective  waiver  management  reduced  the  ability  to  mitigate  airfield 
obstructions.  The  team  also  found  that  Air  Combat  Command  (ACC)  was  using  a  GIS- 
based  Airfield  Obstruction  Management  System  (AOMS)  that  greatly  enhanced  the 
management  of  the  airfield  waiver  program.  At  that  time,  Pacific  Air  Forces  (PACAF) 
had  also  adapted  the  AOMS.  The  team  recommended  that  AF/ILE  endorse  the  AOMS 
for  all  MAJCOMs  (Air  Force  Tiger  Team,  2000,  p.  18).  Prior  to  using  AOMS,  the  most 
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common  method  of  tracking  airfield  obstructions  was  through  the  use  of  Computer-Aided 
Drafting  and  Design  (CADD)  drawings  and  Microsoft  Word,  Microsoft  Excel,  or  paper 
files.  Although  airfield  obstruction  management  systems  will  be  discussed  later,  it  is 
necessary  to  provide  some  background  concerning  the  airfield  environment  and  airfield 
obstruction  classification. 


Airfield  Environment  and  Obstruction  Criteria 

The  airfield  environment  (Figure  2)  consists  of  actual  and  imaginary  surfaces  that 
define  the  obstacle  free  zone  around  the  airfield  (Air  Force  Tiger  Team,  2000,  p.  29). 


E  INNER  HORIZONTAL  SURFACE  (150-feet 
Elevation) 

F  CONICAL  SURFACE  (20: 1  Slope  Ratio) 

G  OUTER  HORIZONTAL  SURFACE  (500-feet 
Elevation) 

H  TRANSITIONAL  SURFACE  (7:1  Slope  Ratio) 

Figure  2.  Airfield  Surfaces  (Adapted  from  Airfield  Obstruction 
Reduction  Initiative  Report,  Nov  2000) 
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The  imaginary  surfaces  (Figure  3)  for  DoD  airfields  are  “the  primary  surface,  the 
approach-departure  clearance  surface,  the  transitional  surface,  the  inner  horizontal 
surface,  the  conical  surface  (fixed-wing  only),  and  the  outer  horizontal  surface  (fixed- 
wing  only)”  ("  Unified  Facilities  Criteria  ",  2001,  p.  217).  An  airfield  obstacle  is  defined 
as  “all  fixed  objects  located  within  the  airfield  environment  that  extend  above  any  of  the 
imaginary  surfaces  of  the  airfield  or  are  located  within  the  mandatory  zone  of  frangibility. 
Airfield  obstacles  may  be  of  either  standard  or  nonstandard  design.  Obstructions  are  also 
classified  as  obstacles”  (AFCESA/CES,  2001,  p.  3).  An  obstruction  can  be  natural  or 
man-made  objects  that  violate  airfield  clearances  (Air  Force  Tiger  Team,  2000,  p.  4). 


IMAGINARY  SURFACES 


DAVIS-MONTHAN  AIR  FORCE  BASE 


TUCSON.  ARIZONA 


CLEAR  ZONE  SURFACE 
APZ\ 

A PI  II 

PRIMARY  SUHFACt 
TRANSITIONAL  SURFACE 

APPROACH.DFPARTURE  SURFACE 
*NNER  HORIZONTAL  SURFACE 


Figure  3.  Imaginary  Surfaces  (Adapted  from  Airfield 
Obstruction  Initiative  Report,  2000) 
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Obstructions  can  be  objects  such  as  signs,  towers,  buildings,  landforms,  trees  and  other 
vegetation  that  “penetrate  regulatory  airspace  surfaces”(CH2M  Hill,  n.d.).  Frangibility  is 
defined  as  “the  ability  of  an  object  to  collapse  or  fall  over  when  struck  by  a  moving 
aircraft  such  as  to  cause  damage  to  aircraft,  not  impede  the  motion  of  the  aircraft,  or 
radically  alter  the  path  of  the  aircraft”  (AFCESA/CES,  2001,  p.  3).  The  frangibility  zone 
consists  of  all  areas  within  250  feet  of  the  runway  center  line  along  its  entire  length  out  to 
a  distance  of  3000  feet  beyond  the  runway  threshold  or  to  the  base  boundary.  It  also 
includes  a  200  foot  lateral  distance  from  all  taxiway  centerlines  (AFCESA/CES,  2001,  p. 
4).  Air  Force  policy  mandates  that  all  obstructions  within  the  primary  surface  and  clear 


The  Zone  of  Frangibility  extends 
76.2  m  (250')  either  side  of  the 
runway  centerline  from  Clear 
Zone  end  to  Clear  Zone  end.  The 
same  requirements  apply  within 
60.96  m  (200')  of  all  taxiway 
centerlines. 


Legend 


Clear  Zone  Boundary 


Area  of  Frangibility 


Figure  4.  Frangibility  Zone  (Adapted  from  AFCESA  ETL,  2001) 


zone  be  identified  as  either  a  permissible  deviation  or  exemption.  A  permissible 
deviation  can  be  visual  and  navigational  aid  facilities  necessary  for  airfield  operations. 
Pennissible  deviations  can  also  be  made  frangible,  if  possible.  An  obstruction  exemption 
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is  identified  as  objects  or  facilities  constructed  before  1964.  A  waiver  is  required  for  any 
obstruction  that  is  not  identified  as  a  permissible  deviation  or  exemption.  A  waiver  can 
be  temporary  or  permanent  and  is  intended  for  those  situations  when  compliance  with  the 
standards  cannot  be  achieved  (Air  Force  Tiger  Team,  2000,  p.  4). 

Airfield  Obstruction  Management  Systems 

A  GeoBase  application  “is  a  stand-alone  application  designed  to  access  the 
GeoBase  service  to  display  a  map,  query  mission  data  via  the  map  or  perfonn  specific 
spatial  analysis  functions  using  geospatial  data”  (HAF  GIO,  2003b.  p.  5).  Both  airfield 
obstruction  management  systems  in  use  by  the  AF  are  GeoBase  applications.  The  two 
systems  in  use  are  the  Airfield  Obstruction  Management  System  (AOMS)  and  the 
Airfield  Obstruction  Tracking,  Analysis,  and  Management  System  (AIROBS).  Both 
systems  were  developed  to  facilitate  the  documenting,  mapping,  analysis,  tracking,  and 
management  of  airfield  obstructions("AIROBS:  Airfield  Obstruction  Management 
System",  2004;  HB&A,  2004). 

AOMS  was  developed  by  Higganbotham,  Briggs,  and  Associates  (HB&A).  The 
AOMS  concept  began  after  HB&A  personnel  visited  ACC  headquarters  in  1997.  Based 
on  the  paper  process  being  used  to  track  and  manage  obstructions,  HB&A  started  the 
development  of  an  electronic  process  for  this  purpose.  HB&A  started  the  development 
without  any  “prompting  or  direction  from  the  AF”  (Mael,  2005).  The  first  version  of 
AOMS  used  in  the  field  came  in  October  1998.  ACC  implemented  the  application  as  a 
pilot  project  at  Davis-Monthan  AFB,  Shaw  AFB,  and  Barksdale  AFB.  The  CADD/GIS 
Technology  Center  continues  to  develop  AOMS  and  the  application  is  non-proprietary 
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and  government-owned.  AOMS  version  8.x  complies  with  the  HAF  GIO  GeoBase 
Enterprise  Architecture  Version  LO  (2003),  and  the  USAF  Garrison  Mapping  CONOPS 
Version  2.0  (2003).  AOMS  version  8.x  also  provides  the  ability  to  produce  command- 
specific  reports,  and  the  AORI  report  (Briggs,  2004,  p.  2,  5).  AOMS  is  being  used  at 
approximately  60  military  installations  worldwide  (Mael,  2005).  Figure  5  is  a  screen 
capture  of  the  AOMS  Database  Menu  and  Figure  6  is  a  screen  capture  example  of  the 
AOMS  Edit  Obstructions  view. 


Figure  5.  AOMS  Database  Menu  Screen 
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Obstruction  Data  Entry  [Coordination  Form] 


“fiR  Wright-Patterson  AFB 

Main  Runway 

Obstruction 

Type 


Description  I  Switch  Box 


J  ustif  ication  I  Switch  box  provides  service  to  airfield  systems  and 
cannot  be  relocated. 


W  Show  Images) 

'  Status  | ACTIVE 
Waiver  type  | 

Precaution 
ORM  Ranking  fl5 


□  RM  group  has  determined  a  RAC  of  15  II  E. 


Frangible 
FAA  Coord 
NAVAID 
Off-Base 
Correctable 


MSL  | 

0.00 

v  AGL  j 

0.00 

v  length  | 

Priority  Area  [3  ~  Within  Frangibility  Zone  |N0 


Violations 

Identifier  Control  Surface 


3C 


*r 


| Primary  Surface 


Actual  Surface 


Reference  to  Regulation 


T 


_  Distance  Distance 

from  threshold.  |  from] 

from  threshold. 


Violation  Amount 


from]- 


Corrective  Actions 

Project  #  Description 


Fund  Year 
Fund  Source 
Category 
FIM 


Estimated  Cost  ($K)  [~ 


v  FIM  Area  [” 


T otal  Estimated  Cost  ($K)  [ 


Installation  Coordination 

Office  Name 

31  -I 


Date  crrY>*AGX» 


Approval 

Office_ Name 


~r 


Date  (YYWMV.PP) 


Media 

'•  ■P 

| - 

- Edit - 1 - 1 

Add 

►  ITTb  of  i 

Figure  6.  AOMS  Edit  Obstructions  Screen 


AIROBS  was  developed  by  CH2M  Hill  for  the  United  States  Air  Forces  in 
Europe  (USAFE)  to  assist  with  the  analysis  and  management  of  airfield  obstructions 
(CH2M  Hill,  2003,  p.  19).  USAFE  had  detennined  that  “no  existing  application  could 
meet  its  specifications  and  performance  requirements.”  Therefore,  USAFE  contracted 
with  CH2M  Hill  to  develop  a  new  application  (Moreno,  2005).  AIROBS  was  developed 
to  “perform  three-dimensional  analyses  to  determine  surface  violations,  display  data 
visually,  create  waivers  for  new  obstructions,  and  produce  summary  reports  and  maps  of 
identified  obstructions.”  CH2M  Hill  holds  the  copyright,  but  the  application  is  owned  by 
the  AF  (CH2M  Hill,  2003,  p.  19-20).  AIROBS  is  in  use  at  approximately  12  AF 
installations  world-wide  (CH2M  Hill,  2004).  Figure  7  is  a  screen  capture  of  the  AIROBS 
Tools  Menu  and  Figure  8  is  a  screen  capture  of  the  AIROBS  Edit  Obstructions  view. 
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gS  airobs 

#  ^  xj- 


_ 52@@ 

Home  Setup  Tools  Help 

Edit  Obstructions 

Query  Obstructions 

Analyze  Obstructions 

Forms  and  Reports 
Export  Obstruction  Data 

Exit 


Figure  7.  AIROBS  Tools  Menu 


Figure  8.  AIROBS  Edit  Obstructions  Screen 
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Common  Characteristics  of  Both  Systems 

According  to  Mr  Michael  Ates,  Civil  Engineer  Division  at  the  Air  Force  Civil 
Engineer  Support  Agency  (AFCESA),  “AOMS  and  AIROBS  are  essentially  the  same  but 
refined  by  two  different  companies”  (Ates,  2005).  Both  applications: 

1.  Use  ESRI  ArcGIS  8.x  software  as  its  platform 

2.  Displays  the  installation  CIP 

3.  Analyzes  obstructions 

4.  Tracks  and  reports  airfield  waivers 

5.  Use  Microsoft  Access  as  the  database 

6.  Have  database  information  organized  by  the  SDSFIE  (Briggs,  2004  p.  2-3;  CH2M 
Hill,  2004) 

Both  applications  are  available  for  AF  use  and  can  be  downloaded  from  the  CADD/GIS 
Technology  Center’s  website.  According  to  the  CADD/GIS  Technology  Center  Project 
#02.030,  both  applications  in  their  final  form  were  posted  to  the  Center’s  website  in 
September  2004.  Also,  the  project’s  timeline  reveals  that  the  decision  to  support  both 
programs  was  made  by  the  AF  in  December  2003  (Homer,  2004).  The  applications  are 
owned  by  the  AF  but  the  copyrights  are  held  by  their  respective  company.  Since  each  of 
the  applications  are  dependent  on  Microsoft  Access  for  the  database  and  the  ESRI 
ArcGIS  platform,  there  is  substantial  cost  involved  every  time  Access  or  ArcGIS  is 
upgraded. 
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Literature  Review  Summary 

The  chapter  began  with  an  overview  and  condensed  history  of  IS  and  GIS.  Then, 
the  background  of  the  GeoBase  program  was  presented  in  order  to  establish  a  basis  for 
airfield  obstruction  management  systems.  Next,  a  discussion  of  the  airfield  surfaces, 
airfield  obstructions,  and  the  AORI  was  provided.  This  discussion  established  the 
foundation  to  the  establishment  of  airfield  obstruction  management  systems.  The  chapter 
closes  with  a  profile  of  the  two  airfield  obstructions  systems  in  use  across  the  AF  today. 
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III.  Methodology 


Overview 

This  research  was  conducted  using  a  qualitative  methodology.  Qualitative 
research  can  use  various  data  collection  and  analysis  methods  (Schwab,  2005).  The 
AFOTEC  Software  Usability  Evaluation  assesses  the  usability  of  a  software  application 
by  identifying  problem  areas  within  the  functions  of  the  application.  The  EUCS 
evaluation  uses  a  numerical  scale  for  the  item  responses  to  determine  the  level  of 
satisfaction  provided  by  the  software  application  to  the  end-user.  The  scale  data  are 
“nominal”  because  the  numbers  represent  categories  (Alreck,  2004).  The  categories 
range  from  Almost  Never  ( 1)  to  Almost  Always  (5).  This  chapter  describes  each 
assessment  and  the  data  collection  methodology.  A  discussion  of  the  evaluation 
instrument  along  with  details  about  the  sample  and  data  analysis  is  presented. 

AFOTEC  Software  Usability  Evaluation 

The  AFOTEC  Software  Usability  Evaluation  focuses  on  the  user’s  interaction 
with  a  software-intensive  system  (HQ  Air  Force  Operational  Test  and  Evaluation  Center, 
1994,  p.  26).  Usability  has  been  described  as  the  leamability,  functionality,  effectiveness, 
acceptability,  and  ease-of-use  of  a  software  application  or  product.  There  are  many 
views  on  the  definition  of  usability.  Bevan,  et  al,  defines  usability  as  “the  ease  of  use  and 
acceptability  of  a  system  or  product  for  a  particular  class  of  users  carrying  out  specific 
tasks  in  a  specific  environment”  (Bevan  et  al,  1991).  The  International  Organization  for 
Standardization  (ISO)/Intemational  Electrotechnical  Commission  (IEC),  9126-1  (2000) 
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defines  usability  as  “the  capability  of  the  software  product  to  be  understood,  learned, 
used  and  attractive  to  the  user,  when  used  under  specified  conditions”  (Bevan,  2003). 
Along  with  the  many  definitions  of  usability,  there  are  many  attributes  associated  with 
the  usability  construct.  Some  of  those  attributes  include  learnability,  understandability, 
reliability,  and  satisfaction,  just  to  name  a  few  (Bevan,  2003;  Juristo  et  al.,  2003).  The 
AFOTEC  Software  Usability  Evaluation  identifies  the  attributes  of  error  abatement, 
responsiveness,  descriptiveness,  consistency,  and  simplicity  (HQ  Air  Force  Operational 
Test  and  Evaluation  Center,  1994,  p.  148)  Table  1  identifies  the  AFOTEC  Software 
Usability  Evaluation  attributes  and  their  definitions. 


Table  1.  Usability  Attributes 


AFOTEC  Usability  Attributes  and  Definitions 

Error 

Abatement 

Aids  in  validating  data  and  avoiding  or  correcting  errors 

Responsiveness 

Allows  the  user  to  effectively  direct  system  operation 

Descriptiveness 

Provides  the  user  with  adequate  explanations  of  every  function 
he/she  is  required  to  perform  and  every  function  the  system  performs 

Consistency 

The  behavior  of  the  software  corresponds  to  the  expectations  of  the 

user 

Simplicity 

Information  presented  to  the  user  is  grouped  into  short,  readily 
understandable  structures 

As  stated  earlier,  the  applications  that  are  the  focus  of  this  research  are  already  in 
use.  Since  these  applications  are  already  being  used,  it  was  necessary  to  find  an 
evaluation  methodology  in  which  the  results  could  be  used  to  make  improvements  to  the 
applications  in  the  field.  After  reviewing  several  methods  of  evaluating  the  usability  of 
software  applications,  such  as  heuristic  evaluation  and  cognitive  walkthroughs,  the 
researcher  detennined  that  the  AFOTEC  Software  Usability  Evaluation  methodology 
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could  be  adapted  for  this  comparative  analysis.  This  methodology  focuses  on  identifying 
problem  areas  with  functions  of  the  applications  which  would  be  effective  for  this  type  of 
comparative  analysis.  Dumas  and  Redish  (1993)  suggest  that  even  though  there  are  many 
ways  to  conduct  usability  testing,  there  are  five  common  characteristics. 

1 .  The  primary  goal  is  to  improve  the  usability  of  the  product. 

2.  The  participants  represent  real  users. 

3.  The  participants  do  real  tasks. 

4.  Observation  and  recording  of  what  participants  do  and  say. 

5.  Analyze  the  data,  diagnose  the  real  problems,  and  recommend  changes  to  fix 
those  problems  (Dumas  &  Redish,  1993,  p.  22). 

Even  though  this  software  evaluation  was  conducted  with  applications  that  are  already  in 
use,  the  characteristics  above  still  apply.  The  AFOTEC  Software  Usability  Evaluation 
instrument  and  process  was  developed  from  the  source  documents  of  Smith  and  Mosier 
(1986),  Ravden  and  Johnson  (1989),  Military  Standard  (MIL-STD)-1472D,  and  MIL- 
STD-1801.  AFOTEC  began  by  testing  and  evaluating  major  weapons  systems,  but  grew 
to  include  the  evaluation  of  the  operator-software  interface  that  has  become  so  important 
in  the  systems  that  the  AF  uses.  The  AFOTEC  evaluation  instrument  is  based  on  a  subset 
of  questions  from  177  evaluation  items  covering  the  attributes  described  earlier  (HQ  Air 
Force  Operational  Test  and  Evaluation  Center,  1994).  The  35  items  used  in  this  research 
to  measure  software  usability  were  selected  by  the  researcher  after  reviewing  both 
applications  and  based  on  the  applicability  of  the  evaluation  items  to  the  tasks  being 
performed  in  the  scenarios.  The  selected  items  were  subsequently  reviewed  and 
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approved  by  the  researcher’s  three  committee  members  who  have  extensive  knowledge 
and  experience  with  software  applications.  The  next  area  for  discussion  is  the  EUCS 
methodology. 

End-User  Computing  Satisfaction 

Prior  to  the  development  of  Doll  and  Torkzadeh’s  EUCS  instrument,  measures  of 
user  information  satisfaction  primarily  focused  around  a  data  processing  environment  or 
overall  computer  user  satisfaction.  Doll  and  Torkzadeh’s  primary  goal  in  the 
development  of  the  EUCS  instrument  was  to  “focus  on  satisfaction  with  the  information 
product  provided  by  a  specific  application  (Doll  &  Torkzadeh,  1988,  p.  260).  They  also 
conceptualized  EUCS  as  the  “affective  attitude  towards  a  specific  computer  application 
by  someone  who  interacts  with  the  application  directly”  (Doll  &  Torkzadeh,  1988,  p. 

261).  The  EUCS  construct  contains  five  subscales  (content,  accuracy,  format,  ease  of 
use,  and  timeliness).  The  EUCS  instrument  used  to  measure  overall  satisfaction  and  the 
five  subscales  consists  of  12  items.  The  instrument  has  been  validated  in  several  past 
studies  by  Doll  and  Torkzadeh  (1988);  Torkzadeh  and  Doll  (1991);  McHaney  and  Cronan 
(1998);  and  Abdinnour-Hehn,  et  al  (2005)  (Abdinnour-Helm  et  al.,  2005;  Doll  & 
Torkzadeh,  1988;  McHaney  &  Cronan,  1998;  Torkzadeh  &  Doll,  1991).  The  internal 
consistency  reliability  (Cronbach’s  alpha)  for  the  overall  EUCS  instrument  used  in  the 
previous  studies  was  .92,  .94,  .91,  and  .94  respectively.  For  the  purpose  of  this  research 
the  subscales  of  accuracy  and  timeliness  were  not  assessed.  Accuracy  of  the  data  used  by 
the  application  could  not  be  assessed  because  the  database  information  was  borrowed 
from  two  active  CE  organizations.  These  organizations  loaded  the  data  as  it  pertained  to 
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their  airfields.  Therefore,  the  researcher  had  no  control  over  the  accuracy  of  the  provided 
data.  Also,  timeliness  was  not  measured  because  the  borrowed  databases  were  similar  in 
the  data  provided  for  the  assessment,  but  were  not  the  same  size.  The  difference  in  size 
could  affect  the  application’s  speed  in  retrieving  the  necessary  data.  Therefore, 
measuring  the  timeliness  of  each  application’s  retrieval  of  data  would  be  inappropriate. 
The  resulting  instrument  used  in  this  research  had  eight  items  measuring  content,  format, 
and  ease  of  use.  The  original  EUCS  instrument  items  and  the  adapted  items  are  shown  in 
Table  2.  The  internal  consistency  measure  for  the  adapted  instrument  was  .76.  A  value 
of  .70  (Cronbach’s  alpha)  is  considered  acceptable  (Nunnally  &  Bernstein,  1994). 


Table  2.  List  of  Original  Questions  and  Current  Questions  Used  for  this  Research 


Subscale 

Original  Question 

Modified  Question 

Does  the  system  provide  the 
precise  information  you  need? 

Does  the  application 
provide  the  precise 
information  you  need? 

Content 

(Doll  &  Torkzadeh,  1988,  p. 
268) 

Does  the  information  content 
meet  your  needs? 

Does  the  information 
content  meet  your  needs? 

Does  the  system  provide 
reports  that  seem  to  be  just 
about  exactly  what  you  need? 

Does  the  application 
provide  reports  that  seem 
to  be  just  about  exactly 
what  you  need? 

Does  the  system  provide 
sufficient  information? 

Does  the  application 
provide  sufficient 
information? 

Format 

(Doll  &  Torkzadeh,  1988,  p. 
268) 

Do  you  think  the  output  is 
presented  in  a  useful  manner? 

Do  you  think  the  output  is 
presented  in  a  useful 
manner? 

Is  the  information  clear? 

Is  the  information  clear? 

Ease  of  Use 

(Doll  &  Torkzadeh,  1988,  p. 
268) 

Is  the  system  user  friendly? 

Is  the  application  user 
friendly? 

Is  the  system  easy  to  use? 

Is  the  application  easy  to 
use? 
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Task  Scenario  Development 

The  AFOTEC  Software  Usability  Evaluation  methodology  and  the  EUCS 
methodology  require  that  users  have  interaction  with  the  applications  in  order  to  conduct 
a  fair  assessment  using  the  instrument  items.  The  evaluators  used  to  conduct  the 
evaluations  had  no  prior  experience  with  the  airfield  obstruction  management  systems 
being  assessed  for  this  research.  The  development  of  task  scenarios  was  necessary  in 
order  for  the  application  evaluators  to  have  some  hands-on  experience  with  the  GIS 
applications.  The  interaction  in  completing  the  task  scenarios  provided  the  background 
to  assist  the  evaluators  in  answering  the  instrument  items.  Prior  to  the  task  scenario 
development,  an  attempt  was  made  to  locate  the  task/requirements  documentation  that  led 
to  the  development  of  each  application.  The  researcher  requested  information  concerning 
the  initial  tasks/requirements  from  HAF  GIO,  ACC,  and  PACAF.  No  historical 
documentation  that  led  to  the  development  of  either  application  was  provided.  Therefore, 
the  task  scenarios  for  this  research  were  developed  based  on  necessary  functions  of  the 
applications  for  the  effective  management  of  airfield  obstruction  infonnation.  CE 
community  planners  provided  these  functional  requirements  through  email  feedback  and 
telephone  conversations.  The  initial  email  request  for  support  can  be  found  at  Appendix 
A.  The  email  was  sent  to  over  30  AF  personnel  who  had  knowledge  of  and/or  experience 
with  airfield  obstruction  management.  The  email  requested  that  the  personnel  “provide 
the  10  most  important  tasks/requirements  that  an  airfield  obstruction  management  system 
should  provide/perform”.  After  very  little  response  from  the  field,  a  follow-up  email  was 
sent  requesting  support.  The  researcher  received  infonnation  from  only  four  respondents. 
Based  on  this  feedback  and  the  author’s  limited  knowledge  of  the  applications,  five 
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scenarios  were  developed.  The  scenarios  encompass  tasks  such  as  loading  a  new 
obstruction,  editing  an  existing  obstruction,  and  producing  a  summary  report.  An 
example  of  one  of  the  scenarios  used  for  the  AIROBS  application  follows: 

The  Flight  Chief  comes  over  and  wants  you  to  load  a  new  obstruction  that 
the  airfield  manager  has  reported  to  him. 

1 .  Select  Tools,  and  then  select  Analyze  Obstructions  on  the  AIROBS  menu. 

2.  In  the  Analyze  Obstructions  box,  fill  in  the  following  information. 

a.  Select  New  Siting  Analysis 

b.  For  the  purpose  of  this  exercise,  select  any  area  on  the  map  outside 
of  the  Analyzed  Obstructions  box.  Once  a  site  is  selected  the  X 
Coord:,  Y  Coord:,  and  Ground  Elevations  should  auto  fill.  The 
height  of  the  obstruction  must  be  filled  in  to  run  the  analysis. 

c.  Height:  3.0 

d.  UFC  needs  to  be  moved  from  Applicable  Criteria  to  Selected  for 
Analysis 

e.  In  the  Selected  Analysis  area,  check  the  boxes  for  Airspace 
Surface  and  Taxiway/ Apron  clearance. 

f.  Select  the  Analyze  button. 

g.  When  the  Result  Summary  appears,  select  Save  Data  to  DB.  At 
this  time  the  Add  New  Structure  window  opens.  Fill  in  the 
following  information: 

Obstruction  Number:  Use  any  number  between  1-100 
Obstruction  Classification:  Temporary  Waiver 
Waiver  Status:  Pending 
Date:  Today’s  Date 
Construction  Type:  Permanent 

Obstruction  Description  and  Comments:  metal  pole  sticking 
out  of  ground 
Frangible:  No 

h.  Select  the  OK  button 

The  Add  New  Structure  window  closes  and  the  Analyze  Obstructions 
window  appears.  Close  this  window. 

This  is  all  that  is  needed  to  load  the  obstruction  data.  The  Engineering 
Assistant  will  collect  and  complete  the  identifying  information  for  the 
obstruction. 
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The  entire  set  of  scenarios  can  be  found  at  Appendix  B  and  C.  Now  that  the  task  scenario 
development  has  been  discussed,  it  is  necessary  to  explain  the  evaluation  instrument  that 
was  used  to  conduct  the  comparative  analysis  of  the  airfield  obstruction  management 
systems. 

Evaluation  Instrument 

The  HAF  GIO  suggested  that  a  comparative  analysis  be  conducted  with  the  two 
airfield  obstruction  management  systems.  This  research  is  a  combination  of  two  methods 
that  focus  on  the  software  usability  and  end-user  computing  satisfaction  constructs.  The 
software  usability  evaluation  is  a  non-statistical  method  aimed  at  identifying  problem 
areas  within  the  applications.  In  addition  to  the  usability  evaluation,  the  author  decided 
to  measure  the  overall  end-user  satisfaction  of  the  applications.  This  area  was  measured 
using  a  revised  version  of  the  EUCS  instrument  discussed  earlier.  The  resulting 
instrument  consisted  of  48  items  made  up  of  open-ended,  close-ended,  and  some 
demographic  questions. 

Items  1-35  were  selected  from  177  items  found  in  the  AFOTEC  Software 
Usability  Evaluation  guidance.  The  evaluator’s  initial  response  was  to  expresses  his/her 
opinion  on  whether  the  design  feature  was  either  Always  or  Not  Always  present  in  the 
software  application.  The  additional  choices  of  Don ’t  Know  and  Not  Applicable  could 
also  be  selected,  if  appropriate.  Where  a  design  feature  was  lacking  (i.e.,  the  response 
being  Not  Always),  the  evaluator  was  asked  to  provide  an  estimate  of  the  difficulty 
caused  by  the  absence  of  that  feature.  These  ratings  were  expressed  in  tenns  of  the 
impact  on  operational  effectiveness,  ranging  from  Very  Low  to  Very  High.  For  those 
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features  that  had  a  response  of  Not  Always  the  evaluator  was  also  asked  to  provide  an 
example  of  where  or  how  that  feature  was  lacking.  Such  an  example  might  be:  “Menu 
selection  options  are  readily  understandable.”  Table  3  provides  an  example  of  an  item 
from  the  AFOTEC  Software  Usability  Evaluation. 


Table  3.  AFOTEC  Software  Evaluation  Example 


10.  Menu  selection  options  are  readily  understandable. 

ALWAYS 

NOT  ALWAYS 

DON’T  KNOW 

NOT  APPLICABLE 

0 

0 

0 

0 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is 

lacking: 

What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW 

LOW 

MEDIUM 

HIGH 

VERY  HIGH 

0 

0 

0 

0 

0 

The  EUCS  questionnaire  items  were  the  next  area  of  the  instrument.  These  items 
were  close-ended  questions  using  a  five  item  Likert  scale  ranging  from  Almost  Always  to 
Almost  Never.  This  part  of  the  research  was  directed  at  measuring  end-user  satisfaction 
on  the  constructs  of  content,  format,  and  ease  of  use.  This  section  of  the  instrument  was 
analyzed  using  simple  descriptive  statistics. 

The  last  area  of  the  instrument  requested  demographic  information  from  the 
evaluators.  The  demographic  questions  simply  addressed  the  rank,  TIS,  and  experience 
level  with  GIS/GeoBase  applications.  The  complete  evaluation  instrument  can  be  found 
at  Appendix  D. 
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Evaluation  Process  and  Sample 

The  application  evaluations  were  conducted  in  a  small  conference  room  located  in 
the  Civil  Engineer  and  Services  School  (CESS).  Two  personal  computers  were  set  up 
with  AIROBS  loaded  on  one  computer  and  AOMS  loaded  on  the  other.  The  AIROBS 
and  AOMS  applications  and  their  respective  databases  that  were  used  for  the  evaluations 
came  from  active  AF  installations.  ESRI’s  ArcGIS  8.3  provided  the  mapping  capability 
and  was  used  with  both  applications.  ESRI  ArcGIS  is  a  GIS  software  used  for 
visualizing,  managing,  and  analyzing  geographic  data  (ESRI,  2006).  The  scenarios  and 
questionnaire  was  provided  to  the  evaluators  at  the  beginning  of  the  evaluation  process. 
The  evaluators  worked  through  the  scenarios  prior  to  answering  the  questionnaire.  The 
evaluators  were  able  to  review  any  part  of  the  application  while  answering  the 
questionnaire.  The  evaluators  were  assigned  to  an  application  on  an  alternating  basis. 
This  allowed  for  the  researcher  to  get  an  equal  number  of  evaluations  for  each 
application.  There  were  20  evaluators  who  provided  ten  evaluations  for  each  application. 

The  installation  community  planner  is  normally  the  individual  responsible  for 
managing  the  airfield  obstruction  program.  The  community  planner  is  usually  an  AF 
civilian  assigned  to  the  installation  CE  squadron.  The  evaluators  chosen  for  this  research 
consisted  of  CE  officers  and  civilians  who  were  Air  Force  Institute  of  Technology 
(AFIT)  master’s  degree  program  students  or  staff  personnel  from  the  CESS.  This  sample 
was  chosen  because  CE  officers  and  civilians  would  likely  be  the  most  representative 
group  outside  of  community  planners  themselves.  All  evaluators  were  volunteers  who 
were  requested  to  participate  via  email.  The  email  requesting  participant  support  can  be 
found  at  Appendix  E.  This  group  was  selected  based  on  their  knowledge  of  CE 
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organizational  processes.  The  rank  of  the  evaluators  ranged  from  Second  Lieutenant  to 
Major  with  one  GS-14  civilian.  The  majority  of  the  evaluators  carried  the  rank  of 
Captain.  The  average  time-in-service  for  all  evaluators  was  six  years  and  five  months 
with  only  four  months  separating  the  average  time-in-service  for  the  AIROBS  and 
AOMS  evaluators.  Just  over  half  of  the  evaluators  had  worked  with  GIS/GeoBase 
applications,  and  those  evaluators  rated  their  level  of  experience  as  “some”. 

Data  Analysis 

The  objective  of  the  AFOTEC  Software  Usability  Evaluation  was  to  identify 
problem  areas  within  the  features  of  the  application.  The  analysis  is  qualitative  and 
focuses  on  the  Not  Always  responses  and  the  comments  that  accompany  those  responses. 
A  three  step  process  is  used  to  complete  the  analysis.  First,  the  items  receiving  a  Not 
Always  response  are  grouped  together  in  order  of  the  impact  ratings  (Very  High  to  Very 
Low )  assigned  by  each  evaluator.  The  next  step  is  to  group  the  responses  by  attribute  and 
function.  This  will  enable  easier  identification  of  interrelated  problems.  The  last  step  is 
to  review  the  comments  provided  by  the  evaluators  in  order  to  further  identify  problem 
areas  (HQ  Air  Force  Operational  Test  and  Evaluation  Center,  1994,  pg.  148).  Appendix 
F  and  G  identifies  the  items,  for  both  applications,  that  received  Not  Always  responses 
and  the  attribute,  function,  and  comments  associated  with  those  items.  The  results  of  the 
analysis  are  normally  provided  in  narrative  format  that  describe  the  key  application 
usability  problems  and  how  those  problems  relate  to  mission  effectiveness  (HQ  Air  Force 
Operational  Test  and  Evaluation  Center,  1994,  p.  148). 
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The  goal  of  the  EUCS  items  was  to  measure  the  satisfaction  provided  to  the 
evaluators  by  the  application.  As  stated  earlier,  this  was  assessed  using  a  five  item  Likert 
scale  ranging  from  Almost  Never  to  Almost  Always.  The  intent  is  to  provide  the  results  of 
the  analysis  by  identifying  problem  areas  and  positive  aspects  of  the  applications.  The 
descriptive  statistics  of  the  EUCS  items  was  analyzed  using  SPSS  statistical  software. 
Each  application  was  analyzed  independently  and  the  results  will  be  provided  in  Chapter 
IV. 

Summary 

The  intent  of  the  evaluation  is  to  provide  an  overall  assessment  of  the  identified 
problem  areas  and  also  assess  the  satisfaction  provided  by  the  application  to  the  end- 
users.  The  evaluation  instrument  was  made  using  AF  software  evaluation  techniques 
along  with  an  adapted  version  of  the  validated  EUCS  instrument.  A  total  of  20  evaluators 
were  used  to  assess  the  AIROBS  and  AOMS  applications,  10  evaluators  for  each, 
respectively.  In  Chapter  IV  the  data  from  the  assessment  is  analyzed  according  to  this 
methodology.  Chapter  V  presents  final  conclusions  and  the  author’s  recommendations  as 
a  result  of  this  research. 
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IV.  Results  and  Analysis 


The  focus  of  this  chapter  is  to  present  the  results  and  analysis  of  the  data  collected 
from  the  assessment  of  the  airfield  obstruction  management  system  applications.  The 
analysis  for  each  application  will  be  presented  within  the  context  of  the  research 
questions.  The  results  of  the  AFOTEC  Software  Usability  Evaluation  will  be  given  using 
a  narrative  format.  The  narrative  will  describe  the  key  software  interface  features  most 
frequently  rated  as  problems  and  the  impact  of  those  design  features  on  operational 
effectiveness  as  identified  by  the  evaluators.  Operational  effectiveness  refers  to  the 
application’s  ability  to  accomplish  the  necessary  tasks.  The  initial  response  by  an 
evaluator  indicated  whether  the  design  feature  was  either  Always  ox  Not  Always  present 
in  the  application.  The  problem  areas  were  identified  by  grouping  the  Not  Always 
responses.  Where  a  design  feature  was  lacking  (i.e.,  an  evaluator  responded  with  Not 
Always),  the  evaluator  provided  an  estimate  of  the  difficulty  caused  by  the  absence  of  that 
feature.  These  estimates  ranged  from  Very  Low  to  Very  High  and  were  used  to  rate  the 
overall  impact  of  that  feature  on  operational  effectiveness.  In  addition,  when  an  item 
received  a  Not  Always  response,  the  evaluator  then  provided  comments  to  support  the 
response.  The  narratives  in  this  chapter  will  focus  on  the  impact  ratings  of  Very  High, 
High,  and  Medium,  where  two  or  more  evaluators  made  comments  about  similar  system 
functions  or  features  that  were  lacking  in  the  application.  For  a  complete  breakdown  of 
the  evaluator  statements  and  impact  ratings  please  see  Appendices  F  and  G.  Although 
identification  of  software  interface  problem  areas  is  the  focus  of  the  research,  the 
researcher  has  also  identified  evaluation  items  which  received  Always  responses  by  a 
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majority  of  the  evaluators.  This  information  could  be  used  to  identify  favorable  design 
features.  The  EUCS  instrument  focuses  on  the  satisfaction  with  the  infonnation  as 
provided  by  an  application  (Doll  &  Torkzadeh,  1988,  p.  260).  Doll  and  Torkzadeh  also 
conceptualized  EUCS  as  the  “affective  attitude  towards  a  specific  computer  application 
by  someone  who  interacts  with  the  application  directly”  (Doll  &  Torkzadeh,  1988  p. 

261).  The  original  EUCS  construct  contains  five  subscales  (content,  accuracy,  format, 
ease  of  use,  and  timeliness).  The  EUCS  is  representative  of  the  underlying  component  of 
these  five  subscales  (Abdinnour-Helm  et  al.,  2005  p.  350).  For  the  purpose  of  this 
research  the  subscales  of  accuracy  and  timeliness  were  not  assessed.  Accuracy  of  the 
data  used  by  the  application  could  not  be  assessed  because  the  database  information  was 
borrowed  from  two  active  CE  organizations.  These  organizations  loaded  the  data  as  it 
pertained  to  their  airfields.  Therefore,  the  researcher  had  no  control  over  the  accuracy  of 
the  provided  data.  Also,  timeliness  was  not  measured  because  the  borrowed  databases 
were  similar  in  the  data  provided  for  the  assessment,  but  were  not  the  same  size.  The 
difference  in  size  could  affect  the  application’s  speed  in  retrieving  the  necessary  data. 
Therefore,  measuring  the  timeliness  of  each  application’s  retrieval  of  data  would  be 
inappropriate.  The  resulting  instrument  used  in  this  research  had  eight  items  measuring 
content,  format,  and  ease  of  use.  The  EUCS  questionnaire  used  a  five  item  Likert  scale 
ranging  from  Almost  Never  (1),  Some  of  the  Time  (2),  About  Half  of  the  Time  (3),  Most  of 
the  Time  (4),  and  Almost  Always  (5).  The  EUCS  results  will  be  discussed  based  on  the 
descriptive  statistics  of  the  EUCS  data.  The  results  discussion  will  begin  with  the  first 
research  question. 
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Research  Question  One  (AOMS) 

How  does  AOMS  rate  in  software  usability,  where  usability  criteria  measures 
error  abatement,  responsiveness,  descriptiveness,  consistency,  and  simplicity? 

The  results  of  the  AFOTEC  Software  Usability  Evaluation  as  applied  to  the 
AOMS  application  indicated  a  few  problems  in  the  usability  attribute  areas  of 
responsiveness,  descriptiveness,  and  simplicity.  There  were  no  overarching  problems 
identified  with  the  error  abatement  nor  consistency  attributes. 

Responsiveness 

As  for  the  responsiveness  attribute,  there  was  only  one  feature  which  received 
negative  comments  with  impact  ratings  from  Very  High  to  Medium.  Several  evaluators 
identified  that  opening  the  GIS  feature  from  the  AOMS  Launcher  took  too  long. 
Otherwise,  the  responsiveness  attribute  was  rated  well  overall.  Of  the  six  evaluation 
items,  five  of  the  items  were  identified  by  a  majority  (six  or  more)  of  evaluators  as 
Always  having  the  design  feature  represented  by  the  evaluation  item.  For  example,  item 
5  ( Text  inputs  are  easy  to  edit)  received  an  Always  response  from  8  of  10  evaluators. 

Descriptiveness 

The  descriptiveness  attribute  received  the  largest  number  of  High  impact  ratings 
on  operational  effectiveness.  Within  the  descriptiveness  attribute  there  were  four 
negative  comments  which  pertained  to  a  lack  of  field  definition  guidance  and  five 
negative  comments  related  to  the  manipulation  of  window  overlays.  The  evaluator’s 
comments  indicated  that  some  of  the  field  definitions  were  difficult  to  understand.  For 
example,  within  the  Status  field  there  were  many  options  on  the  pull  down  menu  that 
would  be  difficult  to  understand  without  knowledge  of  airfield  tenninology.  With 
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regards  to  the  window  overlays  (multiple  windows  of  various  size  that  reside  on  top  of 
other  windows),  the  comments  revealed  that  the  window  overlays  had  to  be  manipulated 
by  minimizing  or  by  moving  an  interfering  window  to  another  area  on  the  screen  in  order 
to  see  the  obstruction  on  the  underlying  map  window.  Of  the  nine  evaluation  items 
representing  the  descriptiveness  attribute,  seven  of  the  items  were  identified  by  a  majority 
(six  or  more)  of  evaluators  as  Always  having  the  design  feature  represented  by  the 
evaluation  item.  For  example,  item  4  {Data  fields  are  adequately  labeled)  received  an 
Always  response  from  8  of  10  evaluators. 

Simplicity 

The  simplicity  attribute  received  comments  with  impact  ratings  ranging  from 
High  to  Very  Low.  The  features  that  received  negative  comments  pertained  to  display 
colors  and  button  locations.  For  example,  the  light  blue  dot  used  to  identify  the 
obstruction  was  difficult  to  locate  on  the  ArcView  map.  The  buttons  are  used  instead  of 
having  to  locate  options  on  the  pull  down  menus.  With  regards  to  the  button  feature,  it 
was  noted  that  the  Analyze  Location  button  and  function  was  difficult  to  understand.  All 
nine  of  the  evaluation  items  received  an  Always  response  from  a  majority  of  the 
evaluators. 

Summary 

The  number  of  Always  responses  received  by  each  item  can  be  found  at 
Appendix  H.  Also,  the  complete  list  of  AOMS  usability  statements  that  received  Not 
Always  responses  along  with  the  evaluators’  comments  can  be  found  at  Appendix  F.  Out 
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of  the  35  AFOTEC  Software  Usability  Evaluation  items  there  were  five  items  that  did  not 
receive  an  Always  response  by  a  majority  (six  or  more)  of  evaluators  (Table  1). 


Table  4.  AOMS  Items  That  Did  Not  Receive  an  Always  Response 
_  by  a  Majority  of  Evaluators _ 


Item  # 

Attribute 

Item 

7 

Responsiveness 

The  system  provides  quick,  positive  feedback  on  the  acceptance 
or  rejection  of  data  entry. 

8 

Error 

Abatement 

The  system  validates  user  inputs  before  processing  them. 

9 

Error 

Abatement 

The  system  provides  adequate  notification  when  it  detects  a  data 
entry  error. 

24 

Descriptiveness 

Window  overlays  are  situated  so  that  they  do  not  obscure 
important  information. 

31 

Descriptiveness 

The  system  provides  adequate  feedback  when  an  internal  fault  is 
detected. 

Research  Question  One  (AIROBS) 

How  does  AIROBS  compare  in  software  usability,  where  usability  criteria 
measures  error  abatement,  responsiveness,  descriptiveness,  consistency,  and  simplicity? 

The  results  of  the  AFOTEC  Software  Usability  Evaluation  as  applied  to  the 
AIROBS  application  indicated  a  few  problems  in  the  usability  attribute  areas  of 
descriptiveness,  simplicity,  and  responsiveness.  There  were  no  overarching  problems 
identified  with  the  error  abatement  nor  consistency  attributes. 

Responsiveness 

Within  the  responsiveness  attribute  the  comments  revealed  a  general  concern  for 
the  system  response  to  data  inputs  and  the  system  speed  when  using  the  zoom  feature. 
For  example,  one  comment  stated,  “When  zooming,  screen  refresh  was  choppy  and 
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sluggish.”  Of  the  six  evaluation  items,  five  of  the  items  received  an  Always  response 
from  a  majority  of  the  evaluators. 

Descriptiveness 

The  descriptiveness  attribute  received  the  most  Very  High  to  Medium  operational 
effectiveness  impact  ratings.  The  majority  of  the  comments  received  by  the 
descriptiveness  attribute  pertained  to  the  obstruction  identifier,  window  overlays,  and 
data  fields.  Based  on  the  comments  provided  by  the  evaluators  it  seemed  that  the 
evaluators  experienced  difficulty  in  locating  the  obstruction  on  the  ArcView  Map  after  it 
had  been  selected  from  the  Edit  Obstructions  list.  Also,  there  were  comments  about  the 
acronyms  and  drop  down  menus  used  for  data  field  inputs.  The  acronyms  [FIM  (Facility 
Index  Matrix),  etc.]  and  selections  from  the  drop  down  menus  could  be  confusing  for  an 
airfield  manager  who  has  access  to  the  application.  The  largest  amount  of  comments  was 
centered  on  the  window  overlays.  Of  the  ten  evaluators,  nine  provided  negative 
comments  to  item  24  of  the  evaluation.  This  item  received  impact  ratings  ranging  from 
Medium  to  Very  High.  The  evaluators  found  that  the  window  overlays  were  difficult  to 
manipulate  in  order  to  view  necessary  information  on  underlying  windows.  Some  of  the 
windows  had  to  be  closed  in  order  to  access  previously  opened  windows.  Of  the  nine 
descriptiveness  items,  seven  of  the  items  were  identified  by  a  majority  (six  or  more)  of 
evaluators  as  Always  having  the  design  feature  represented  by  the  evaluation  item. 

Simplicity 

The  simplicity  attribute  received  impact  ratings  that  ranged  from  Very  High  to 
Low.  The  simplicity  attribute  refers  to  the  information  being  grouped  and  presented  to 
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the  user  in  short,  readily  understandable  structures.  The  items  that  received  the  highest 
impact  ratings  pertained  to  the  lack  of  ease  when  trying  to  locate  infonnation  on  the 
screen,  general  difficulty  understanding  data  field  options,  and  difficulty  with  the  window 
overlays.  The  comments  received  about  finding  information  on  the  screen  and  the 
difficulty  with  the  window  overlays  were  very  similar  to  the  comments  received  for  those 
same  features  described  previously  within  the  descriptiveness  attribute  narrative.  Again, 
the  evaluators  had  difficulty  finding  the  selected  obstruction  and  it  was  not  easy  to 
manipulate  the  windows  when  several  were  open  at  the  same  time.  One  comment 
concerning  the  window  overlays  was,  “I  had  difficulty  with  viewing  map  and  getting  rid 
of  overlays”.  In  addition,  several  evaluators  found  that  the  data  field  options  were 
difficult  to  understand.  An  example  of  a  comment  concerning  the  data  fields  was, 
“Confusion  with  ‘Obstruction  Frangible’  and  ‘Frangible’  field”.  Of  the  ten  items  relating 
to  the  simplicity  attribute,  three  items  received  Not  Always  responses  from  a  majority  of 
the  evaluators. 

Summary 

The  number  of  Always  responses  received  by  each  item  for  the  AIROBS 
application  can  be  found  at  Appendix  H.  In  addition,  the  complete  list  of  AIROBS 
usability  statements  that  received  Not  Always  responses  along  with  the  evaluators’ 
comments  can  be  found  at  Appendix  G.  Out  of  the  35  evaluation  items  there  were  eight 
items  that  did  not  receive  an  Always  response  by  a  majority  (six  or  more)  of  evaluators 
(Table  5).  Three  of  the  eight  items  pertained  to  a  lack  of  guidance  on  data  entry  format 
and  a  lack  of  system  validation  and  feedback.  The  other  items  concerning  ease  of 
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locating  infonnation  and  window  overlays  were  discussed  previously.  The  next  section 
will  address  the  second  research  question. 


Table  5.  AIROBS  Items  That  Did  Not  Receive  an  Always  Response 
_  by  a  Majority  of  Evaluators _ 


Item  # 

Attribute 

Item 

6 

Consistency 

Where  data  are  entered  from  source  documents,  the  format  for 
data  entry  corresponds  to  that  of  the  source  documents. 

7 

Responsiveness 

The  system  provides  quick,  positive  feedback  on  the  acceptance 
or  rejection  of  data  entry. 

8 

Error 

Abatement 

The  system  validates  user  inputs  before  processing  them. 

17 

Simplicity 

Information  is  easy  to  find  on  the  screen. 

24 

Descriptiveness 

Window  overlays  are  situated  so  that  they  do  not  obscure 
important  information. 

25 

Simplicity 

Where  several  windows  are  displayed  simultaneously,  it  is  easy 
for  the  user  to  shift  among  them  to  select  which  window  is  to  be 
made  active. 

28 

Simplicity 

Graphic  symbology  is  appropriate  for  the  information  it 
represents. 

31 

Descriptiveness 

The  system  provides  adequate  feedback  when  an  internal  fault  is 
detected. 

Research  Question  Two 

The  EUCS  questionnaire  items  were  close-ended  questions  which  addressed  the 
content,  format,  ease  of  use,  and  global  satisfaction  with  the  software  application.  The 
EUCS  questionnaire  sought  to  measure  the  satisfaction  with  the  content  and  format  of  the 
information  provided  by  the  application  and  the  overall  ease  of  use  of  the  application. 

The  questionnaire  used  a  five  item  Likert  scale  ranging  from  Almost  Never  (1),  Some  of 
the  Time  (2),  About  Half  of  the  Time  (3),  Most  of  the  Time  (4),  and  Almost  Always  (5). 
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Based  on  this  scale,  a  mean  of  4  (Most  of  the  Time)  or  better  would  indicate  that  the 
evaluators  found  the  application  provided  the  intent  of  the  questionnaire  item  between 
Most  of  the  Time  (4)  and  Almost  Always  (5).  Means  that  are  above  4.5  would  indicate  a 
higher  number  of  evaluators  rated  the  application  as  Almost  Always  in  terms  of  the 
content,  format,  and  ease  of  use  as  specified  by  the  questionnaire  items.  As  stated 
previously  in  the  chapter,  the  EUCS  is  representative  of  the  underlying  components  of  the 
five  subscales  (Abdinnour-Helm  et  al.,  2005,  p.  350).  Although,  for  this  research,  only 
the  subscales  of  content,  format,  and  ease  of  use  was  measured.  The  data  from  the 
questionnaire  was  input  by  the  researcher  and  analyzed  using  SPSS,  a  statistics  software 
application.  The  next  section  will  use  descriptive  statistics  to  discuss  the  results  of  the 
EUCS  questionnaire. 

AOMS 

How  did  AOMS  rate  in  end-user  computing  satisfaction  (EUCS),  as  determined 
by  the  measures  of  content,  format,  and  ease  of  use  provided  by  each  application? 

The  internal  consistency  measure  and  mean  for  the  global  EUCS  construct  as 
applied  to  the  AOMS  application  was  .83  and  4.61  respectively.  The  standard  deviation 
of  the  overall  questionnaire  (3.81)  and  the  content  subscale  (2.37)  reveal  a  higher  level  of 
variation  in  the  responses  for  these  two  areas.  The  reliability  measure  (.83)  indicates 
good  internal  consistency  of  the  items  in  the  scale.  A  value  of  .70  (Cronbach’s  alpha)  is 
considered  acceptable  (Nunnally  &  Bernstein,  1994).  Table  6  provides  the  descriptive 
statistics  for  the  AOMS  application  to  include  results  from  the  content,  format,  and  ease 
of  use  constructs.  The  means  for  the  items  within  the  content  construct  ranged  from  4.5 
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to  4.7  with  an  overall  mean  of  4.65  and  the  internal  consistency  measure  was  .78.  The 
mean  for  the  format  construct  was  4.6  with  an  internal  consistency  measure  of  .86.  The 
ease  of  use  construct  had  a  mean  of  4.55  with  an  internal  consistency  measure  of  .95.  As 
shown  in  Table  6,  the  means  for  the  subscales  and  the  overall  AOMS  EUCS  construct 
range  from  4.55  to  4.65.  The  ratings  received  for  the  subscales  and  the  overall 
questionnaire  as  applied  to  the  task  scenarios  revealed  that  the  aspects  of  the  application 
(content,  format,  ease  of  use)  highly  influenced  the  satisfaction  of  the  evaluators. 


Table  6.  AOMS  EUCS  Statistics 


Item/Subscale 

D 

M 

SD 

Overall  EUCS  Statistics  for  AOMS 

.83 

4.61 

3.81 

Content 

.78 

4.65 

2.37 

36.  Does  the  application  provide  the  precise  information  you  need? 

4.70 

.67 

37.  Does  the  information  content  meet  your  needs? 

4.50 

1.10 

38.  Does  the  application  provide  reports  that  seem  to  be  just  about  exactly 
what  you  need? 

4.70 

.48 

39.  Does  the  application  provide  sufficient  information? 

4.70 

.67 

Format 

.86 

4.60 

1.14 

40.  Do  you  think  the  output  is  presented  in  a  useful  manner? 

4.50 

.70 

41 .  Is  the  information  clear? 

4.70 

.48 

Ease  of  Use 

.95 

4.55 

1.37 

42.  Is  the  application  user  friendly? 

4.60 

.70 

43.  Is  the  application  easy  to  use? 

4.50 

.70 

n=10 
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AIROBS 


How  did  AIROBS  rate  in  end-user  computing  satisfaction  (EUCS),  as  determined 
by  the  measures  of  content,  format,  and  ease  of  use  provided  by  the  application? 

The  internal  consistency  measure  and  mean  for  the  questionnaire  as  applied  to  the 
AIROBS  application  was  .62  and  4.43  respectively.  The  standard  deviation  of  the  overall 
questionnaire  (2.72)  reveals  variation  in  the  responses  provided  by  the  evaluators,  but  is 
lower  than  the  standard  deviation  for  AOMS  (3.81).  Based  on  the  reliability  measure  of 
the  AOMS  application  and  other  past  studies  referenced  in  chapter  2,  the  reliability 
measure  (.62)  is  lower  than  desired  when  compared  to  the  AOMS  reliability  measure 
(.83).  According  to  Guilford  (1954),  there  are  no  “hard-and-fast”  rules  to  indicate  how 
high  the  reliability  coefficients  should  be  (Guilford,  1954).  When  a  reliability  measure 
(Cronbach’s  alpha)  “proves  to  be  very  low,  either  the  test  is  too  short  or  the  items  have 
very  little  in  common”  (Nunnally,  1978,  p.  231).  In  this  case,  the  questionnaire  is  short, 
but  the  items  have  been  proven  through  past  research  to  be  internally  consistent.  Table  7 
identifies  the  descriptive  statistics  for  the  AIROBS  application  to  include  the  content, 
format,  and  ease  of  use  constructs.  The  means  for  the  items  within  the  content  construct 
ranged  from  4.4  to  4.8  with  an  overall  mean  of  4.55  and  the  internal  consistency  measure 
was  .67.  The  mean  for  the  format  construct  is  4.4  with  an  internal  consistency  measure 
of  .21.  Asa  rule  of  thumb  an  internal  consistency  measure  “below  .5  would  be 
considered  unacceptable”  (George  &  Mallery,  2005).  The  ease  of  use  construct  had  a 
mean  of  4.2  with  an  internal  consistency  measure  of  .90.  As  shown  in  Table  7,  the  means 
for  the  subscales  and  the  overall  AIROBS  EUCS  construct  range  from  4.2  to  4.55.  The 
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ratings  received  for  the  subscales  and  the  overall  questionnaire  as  applied  to  the  task 
scenarios  revealed  that  the  aspects  of  the  application  (content,  format,  ease  of  use) 
influenced  the  satisfaction  of  the  evaluators. 


Table  7.  AIROBS  EUCS  Statistics 


Item/Subscale 

a 

M 

SD 

Overall  EUCS  Statistics  for  AIROBS 

.62 

4.43 

2.72 

Content 

.67 

4.55 

1.55 

36.  Does  the  application  provide  the  precise  information  you  need? 

4.60 

.52 

37.  Does  the  information  content  meet  your  needs? 

4.40 

.52 

38.  Does  the  application  provide  reports  that  seem  to  be  just  about  exactly 
what  you  need? 

U 

4.40 

.70 

39.  Does  the  application  provide  sufficient  information? 

4.80 

.42 

Format 

.21 

4.40 

1.03 

40.  Do  you  think  the  output  is  presented  in  a  useful  manner? 

4.50 

.71 

41.  Is  the  information  clear? 

4.30 

.67 

Ease  of  Use 

.90 

4.20 

1.51 

42.  Is  the  application  user  friendly? 

4.20 

.63 

43.  Is  the  application  easy  to  use? 

4.20 

.92 

n=10 


Summary 

This  chapter  outlined  the  results  of  the  AFOTEC  Software  Usability  Evaluation 
and  the  EUCS  questionnaire.  The  AFOTEC  evaluation  identified  problem  areas  within 
both  applications  that  could  benefit  the  improvement  of  the  products.  The  EUCS  simply 
used  descriptive  statistics  to  measure  the  constructs  of  content,  format,  and  ease  of  use 
along  with  the  overall  satisfaction  provided  by  the  application.  Since  the  internal 
consistency  of  the  format  construct  for  the  AIROBS  EUCS  assessment  was  so  low,  it  was 
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not  used  in  any  comparative  judgment.  Chapter  V  will  present  conclusions  and 
recommendations  based  on  the  analysis  described  in  this  chapter  and  the  researcher’s 
observations  throughout  this  study. 
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V.  Discussion  and  Conclusions 


The  purpose  of  this  chapter  is  to  present  the  conclusions  that  resulted  from  this 
research  study.  It  will  begin  with  a  brief  discussion  of  the  findings  from  the  comparative 
analysis  of  AOMS  and  AIROBS.  Next,  some  recommendations  of  the  researcher  will  be 
provided.  Then,  the  limitations  discovered  during  this  research  will  be  presented.  The 
chapter  will  close  with  some  suggestions  for  future  research. 

Findings 

Research  Question  One 

How  do  AOMS  and  AIROBS  compare  in  software  usability,  where  usability 
criteria  measures  error  abatement,  responsiveness,  descriptiveness,  consistency,  and 
simplicity? 

The  AFOTEC  Software  Usability  Evaluation  revealed  that  AOMS  revealed  fewer 
usability  problems  than  AIROBS.  The  AOMS  application  received  19  Not  Always 
responses  versus  the  25  received  by  AIROBS.  Also,  AOMS  received  fewer  negative 
comments  (37)  overall,  than  did  AIROBS  (61).  Both  applications  had  problems 
identified  with  the  window  overlays  and  data  fields,  but  AIROBS  received  more  negative 
comments  in  both  areas.  Also,  as  shown  by  Table  8,  there  were  five  items  that  received  a 
significant  difference  of  Always  responses  between  the  two  applications.  A  significant 
difference  means  that  one  application  received  a  majority  of  Always  responses  while  the 
other  application  did  not  receive  a  majority  of  Always  responses  for  the  same  item.  The 
AOMS  application  received  a  majority  of  Always  responses  for  four  of  the  five  items. 
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The  AIROBS  application  only  received  a  majority  of  Always  responses  for  one  of  the  five 
items.  Finally,  as  stated  in  chapter  2,  AOMS  is  being  used  at  approximately  60  military 
installations  worldwide  and  AIROBS  is  in  use  at  approximately  12  AF  installations 
world-wide  (CH2M  Hill,  2004;  Mael,  2005). 


Table  8.  Items  with  a  Significant  Difference  of  Always  Responses 


Item  # 

Attribute 

Item 

#  Always 
AOMS 

#  Always 

AIROBS 

6 

Consistency 

Where  data  are  entered  from  source  documents, 
the  format  for  data  entry  corresponds  to  that  of 
the  source  documents. 

6 

2 

9 

Error 

Abatement 

The  system  provides  adequate  notification  when 
it  detects  a  data  entry  error. 

3 

6 

17 

Simplicity 

Information  is  easy  to  find  on  the  screen. 

7 

5 

25 

Simplicity 

Where  several  windows  are  displayed 
simultaneously,  it  is  easy  for  the  user  to  shift 
among  them  to  select  which  window  is  to  be 
made  active. 

9 

4 

28 

Simplicity 

Graphic  symbology  is  appropriate  for  the 
information  it  represents. 

10 

5 

n=10 


Research  Question  Two 

How  do  AOMS  and  AIROBS  compare  in  end-user  computing  satisfaction 
(EUCS),  as  determined  by  the  measures  of  content,  format,  and  ease  of  use  provided  by 
each  application? 

The  EUCS  analysis  revealed  that  AOMS  had  higher  means  for  overall  EUCS, 
content  and  ease  of  use.  For  the  overall  EUCS,  the  mean  for  AOMS  was  4.61  as  opposed 
to  4.43  for  AIROBS.  The  means  for  the  content  subscale  were  4.65  and  4.55  for  AOMS 
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and  AIROBS,  respectively.  The  ease  of  use  subscale  had  a  mean  of  4.55  for  AOMS  and 
4.2  for  AIROBS. 


Recommendations 

This  research  focused  on  conducting  a  comparative  analysis  of  the  AOMS  and 
AIROBS  applications  using  the  AFOTEC  Software  Usability  Evaluation  and  EUCS 
questionnaire  as  the  framework  for  the  research.  Although,  during  this  research  process 
there  were  some  discoveries  made  which  have  led  to  the  recommendations  that  are 
presented  in  the  next  few  paragraphs. 

Single  Owner  for  Applications 

Currently,  there  is  no  “owner”  for  the  AOMS  or  AIROBS  applications.  “Owner” 
in  this  respect  would  be  the  organization  that  is  the  POC  for  the  application.  The  owner 
would  be  the  keeper  and  maintainer  of  the  requirements  documents  that  led  to  an 
application’s  development.  In  addition,  the  owner  would  be  the  single  POC  for 
customers  to  contact  for  support  and  guidance  and  the  application  developer  would  also 
contact  the  owner  when  new  upgrades  or  versions  are  ready  for  implementation.  The 
owner  would  be  the  sole  representative  of  the  AF  when  conducting  business  with 
application  developers.  It  would  also  be  advantageous  when  fielding  changes  and 
upgrades  to  the  applications  because  the  owner  would  be  responsible  to  ensure  that  all 
users  have  the  same  versions  of  the  applications. 

Application  Registry  and  Development  Process 

The  HAF  GIO  should  adapt  and  mandate  the  use  of  a  GIS  application  registry  and 
an  application  development  process  as  suggested  by  Colby  Free  of  ACC.  Colby  Free  is  a 
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contractor  assigned  to  Headquarters  ACC  Installations  and  Support  division,  where  he 
was  given  the  responsibility  to  draft  an  application  development  process  for  the  HAF 
GIO.  The  purpose  of  the  application  development  process  and  application  registry  is  to 
reduce  redundancy  by  creating  a  process  to  coordinate  and  identify  the  applications  that 
are  being  used  for  specific  purposes  (Free,  2005).  A  GIS  application  registry  would  give 
potential  users/organizations  visibility  to  what  applications  are  already  in  use  for  specific 
missions.  This  would  provide  the  potential  user/organization  the  application  owner  who 
could  be  contacted  for  information  and/or  implementation.  The  registry  would  identify 
the  application  owner  who  could  be  contacted  for  infonnation  about  the  product.  If  an 
application  could  not  be  found  for  a  specific  mission  purpose,  then  the  application 
development  process  could  be  followed  prior  to  purchasing  a  COTS  application  or 
pursuing  the  development  of  a  new  application.  The  registry  would  provide  visibility  to 
the  applications  in  use  and  provide  visibility  to  other  functional  areas  (i.e.,  Security 
Forces,  Communications)  who  may  be  able  to  gain  from  existing  applications  (Free, 
2005). 

Single  Application  for  Single  Purpose 

During  the  research  process  it  was  discovered  that  both  AOMS  and  AIROBS 
were  supported  and  approved  for  use  across  the  AF.  It  would  seem  advantageous  for  the 
AF  to  have  software  application  companies  produce  prototypes  (based  on  sound 
requirements  documents)  to  be  field  tested  for  the  application’s  intended  mission 
purpose.  This  process  could  be  similar  to  the  process  used  by  the  acquisitions 
community  in  order  to  prevent  purchasing  and  maintaining  multiple  similar  applications. 
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Once  the  testing  is  complete,  the  AF  could  select  the  best  application  to  field  across  the 
AF,  as  opposed  to  what  happened  with  AOMS  and  AIROBS.  The  use  of  the  best,  single 
application  could  save  funds  in  software  upgrades  and  also  reduce  funds  spent  on  training 
personnel  to  use  multiple  applications. 

Limitations 

There  were  several  limitations  that  may  have  impacted  this  research.  First  of  all, 
earlier  in  the  research  process,  the  researcher  contacted  community  planners  and 
MAJCOM  Geo  Integration  Officers  in  an  effort  to  locate  the  requirements  documents/list 
that  led  to  the  development  of  AOMS  and  AIROBS.  No  documented  information  could 
be  located.  Due  to  this  lack  of  information  the  task  scenarios  used  by  the  evaluators  may 
not  have  been  as  robust  as  scenarios  developed  from  the  requirements  documents/list. 

Secondly,  this  research  did  not  uncover  an  existing  process  or  validated 
instrument  for  the  purpose  of  conducting  comparative  analyses  of  fielded  software 
applications.  The  usability  evaluation  portion  of  this  research  was  conducted  using  a 
process  that  was  intended  to  be  used  in  the  application  design  process.  Even  though 
every  effort  was  made  by  the  researcher  to  choose  evaluation  items  that  could  be  applied 
to  fielded  applications,  there  were  some  questionnaire  items  that  were  not  applicable  to 
the  applications  evaluated  as  indicated  by  the  responses  from  the  evaluators.  Therefore, 
there  may  have  been  other  AFOTEC  Software  Usability  Evaluation  items  that  could  have 
made  this  research  more  robust  with  regards  to  completeness  of  responses. 

Next,  the  evaluators  had  no  prior  experience  with  automated  airfield  obstruction 
management  systems.  In  addition,  the  evaluators  had  a  limited  amount  of  time  to  spend 
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with  the  applications.  The  task  scenarios  were  developed  to  be  accomplished  in  a  step- 
by-step  manner.  Due  to  this  structure,  the  evaluators  did  not  spend  additional  time 
exploring  other  features  of  the  applications.  Because  of  the  lack  of  experience  with 
airfield  obstruction  management  systems  and  the  limited  time  spent  using  the 
applications;  the  evaluators  may  have  been  “impressed”  by  the  applications  which  could 
have  led  to  the  inflation  of  scores  that  were  recorded  for  the  EUCS  portion  of  the 
evaluation. 

Finally,  there  could  be  an  issue  with  validity  and  reliability  of  the  evaluation.  The 
AFOTEC  Software  Usability  Evaluation  is  primarily  used  in  the  design  process  of 
military  systems  and  software  applications.  It  dates  back  to  1994,  and  has  not  had  any 
recent  updating  of  the  process.  Also,  no  documentation  was  found  which  supported  its 
use  in  the  civilian  sector.  The  EUCS  with  all  five  subscales  (content,  format,  accuracy, 
ease  of  use,  timeliness)  was  validated  and  proved  reliable  in  recent  studies,  but  this 
research  elected  not  to  measure  accuracy  and  timeliness.  Without  these  two  subscales, 
one  must  question  the  validity  and  reliability  of  the  questionnaire  until  additional  testing 
can  be  accomplished.  Overall,  this  is  the  first  known  attempt  to  combine  these  two 
distinct  methodologies  to  conduct  a  research  project.  Therefore,  the  generalizability  of 
this  methodology  will  be  unknown  until  future  testing  of  this  tool. 

Future  Research 

This  was  the  first  time  an  in-depth  comparative  analysis  has  been  conducted  on 
two  GIS  applications  being  used  in  the  AF.  This  research  process  could  be  used  as  a 
benchmark  to  conduct  other  comparative  analyses  against  applications  which  are  similar 
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and  being  used  for  the  same  mission  purpose.  Also,  future  research  could  be  conducted 
to  assess  the  status  and  implementation  of  the  recommended  GIS  application  registry  and 
application  development  process. 
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Appendix  A:  Initial  E-mail  Request  for  Support 


I  am  an  Air  Force  Institute  of  Technology  (AFIT)  student  currently  working  on  a  thesis 
research  project  in  the  area  of  airfield  obstruction  management  systems.  I  am  asking  for 
your  support  in  my  research  effort.  This  research  effort  is  supported  by  Headquarters  Air 
Force  Geo  Integration  Office  (HAF  GIO)  and  the  GIS  Support  Center. 

Each  of  you  was  selected  based  on  either  being  a  user  of  either  AOMS  or  AIROBS,  or 
having  familiarity  with  managing  airfield  obstructions. 

No  personal  identifying  information  will  be  used  in  the  final  research  product.  All 
collected  information  will  be  retained  by  the  researcher. 

Based  on  your  own  experience  and  understanding  of  airfield  obstruction  management 
systems,  please  provide  the  following: 

1.  Without  regard  to  the  system/process  that  you  currently  use,  please  provide  the  10 
most  important  tasks/requirements  that  an  airfield  obstruction  management  system  should 
provide/perfonn.  Please  provide  the  tasks/requirements  in  rank  order  of  importance. 

2.  (Optional)  Please  provide  a  short  scenario  that  could  be  used  to  demonstrate  the 
functionality/usability  of  an  airfield  obstruction  management  system.  The  scenario  can 
be  an  input,  update,  retrieval,  etc.  If  the  scenario  includes  other  documentation  such  as  a 
request  for  waiver,  feel  free  to  send  the  electronic  documentation. 

Please  return  the  requested  infonnation  by  Wednesday,  9  Nov. 

I  sincerely  appreciate  your  support  with  my  research  project.  Please  feel  free  to  contact 
me  or  my  thesis  advisor  (Lt  Col  Summer  Bartczak,  email:  Summer.Bartczak@afit.edu)  if 
you  have  any  questions  regarding  this  request  or  the  research. 


//SIGNED// 

Mark  E.  Barner,  SMSgt,  USAF 
Graduate  Student 
AFIT/ENV 
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Appendix  B:  Airfield  Obstruction  Management  System  (AOMS)  Scenarios 


These  scenarios  have  been  designed  to  give  you  some  hands-on  interaction  with  the 
application.  The  scenarios  will  give  you  some  familiarization  with  a  few  of  the  most 
important  requirements  expected  of  an  airfield  obstruction  management  system.  If  you 
feel  it  is  necessary,  please  take  notes  throughout  the  completion  of  the  scenarios.  Upon 
completion  of  the  scenarios  you  should  be  able  to  complete  the  evaluation  questionnaire. 
You  may  also  review  the  questionnaire  prior  to  or  during  the  completion  of  the  scenarios. 
If  you  think  that  more  interaction  is  needed,  feel  free  to  spend  additional  time  with  the 
application. 

Log  into  the  computer  using  the  following  information: 

User  Name:  Tester 

Password:  XSW@cde3 

Log  on  to:  CEXP33XY  (this  computer) 

Scenario  1 


You’ve  recently  arrived  at  your  new  duty  station,  XYZ  AFB.  You  are  assigned  to  the 
Engineering  Flight  and  the  Flight  Chief  wants  you  to  have  some  familiarity  with  all  of  the 
areas  within  the  flight.  You  are  going  to  be  working  with  the  community  planner  to  get 
some  familiarization  with  that  position.  One  of  the  responsibilities  of  the  community 
planner  is  to  track,  manage,  and  analyze  airfield  obstructions.  There  is  a  fairly  new 
system  in  place  which  assists  in  accomplishing  this  task.  The  Airfield  Obstruction 
Management  System  is  the  program  used  at  XYZ  AFB.  The  application  uses  Microsoft 
Access  as  its  database  and  ESRI  ArcView  8.3  for  its  mapping  capability. 

1 .  Open  AOMS  from  the  desktop. 

2.  Select  the  HELP  button  on  the  AOMS  Launcher  menu. 

3.  Select  the  STARTUP  tab  and  spend  a  few  minutes  reviewing  the  features  of  the 
STARTUP  area. 

4.  When  finished  in  the  STARTUP  area,  select  the  ArcMap  User’s  Manual  from  the 
HELP  system  menu.  Spend  a  few  minutes  reviewing  the  features  of  the  ArcMap 
User’s  Manual. 

5.  When  finished  in  the  ArcMap  User’s  Manual  area,  select  the  Database  User’s 
Manual  from  the  HELP  system  menu.  Spend  a  few  minutes  reviewing  the 
features  of  the  Database  User’s  manual  area. 

6.  When  finished,  close  the  HELP  system. 

You  should  be  back  at  the  AOMS  Launcher  menu. 

Scenario  2 
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The  Flight  Chief  comes  over  and  wants  you  to  load  a  new  obstruction  that  the  airfield 
manager  has  reported  to  him. 

3.  Select  the  GIS  button  on  the  AOMS  Launcher  menu.  This  loads  the  maps. 

4.  You  can  adjust  the  window  size  and  move  the  toolbar. 

5.  Find  and  select  the  Launch  AOMS2000  Database  button  (should  be  4th  button 
from  the  right  on  the  toolbar).  This  opens  the  AOMS  Database  window. 

6.  Select  the  New  Obstruction  button.  You  will  input  a  New  Obstruction  with  the 
following  information: 

7.  The  obstruction  Type  will  be  Waiver.  AOMS  assigns  the  obstruction  Number 
as  the  next  available  number  for  that  obstruction  type  (auto-numbering).  This 
number  can  be  changed  if  desired  to  match  any  existing  numbering  scheme. 
Alphanumeric  values  can  be  used,  but  will  not  be  evaluated  for  obstruction  auto¬ 
numbering.  Fill  in  the  rest  of  the  obstruction  data  with  the  following  infonnation: 
Description:  2  ft.  by  3  ft.  by  2  ft  deep  hole  near  arresting  barrier  shack  on  the 
north  end,  base  side 

Justification:  hole  meets  criteria  to  be  an  obstruction 

Remark:  recommend  that  this  be  repaired  with  an  in-house  work  order 

Status:  Active 

Waiver  type:  Temporary 

Safety  Precaution:  Other 

Frangible:  No 

FAA  Coordination:  No 

NAVAID:  No 

Off-Base:  No 

Correctable:  Yes 

In  the  Violations  area,  Control  Surface  field,  select  None 
Select  the  OK  button 

Once  this  is  done  you  should  be  back  at  the  AOMS  Database  Menu 

This  is  all  that  is  needed  to  load  the  obstruction  data.  The  Engineering  Assistant  will 
collect  the  specific  obstruction  coordinates  and  complete  the  identifying  information  for 
the  obstruction. 

Scenario  3 


It  is  now  at  the  end  of  the  day  and  you  receive  a  call  from  the  Horizontal  shop  foreman. 
He  informs  you  that  the  hole  out  near  the  barrier  shack  has  been  filled  and  the  area 
seeded.  Since  it  is  now  determined  that  the  hole  is  no  longer  an  obstruction  then  the 
AOMS  needs  to  be  updated. 

1 .  From  the  AOMS  Launcher  menu  select  the  DATABASE  button. 

2.  Select  the  Browse  Obstructions  button  from  the  AOMS  database  menu. 

3.  Find  the  obstruction  that  you  loaded  for  the  hole  near  the  barrier  shack. 


57 


4.  Once  the  obstruction  is  located,  open  the  obstruction  information. 

5.  In  the  Status  data  field,  change  the  status  to  Corrected  and  select  the  OK  button. 

6.  Close  the  Browse  Airfield  Obstruction  Records  window. 

7.  You  have  now  updated  the  database  with  the  new  status. 

Scenario  4 


The  Airfield  Manager  has  requested  a  summary  report  of  all  obstructions. 

1 .  Select  the  Reports  tab  on  the  AOMS  Database  Menu 

2.  Select  the  Summary  Report  option,  and  then  select  All  Obstructions. 

3.  When  the  Summary  Report  appears  you  can  Close  the  report  and  Close  the  Build 
Obstruction  Report  window. 

Scenario  5 


An  individual  from  the  Weather  Squadron  is  in  your  office  requesting  some  information 
concerning  one  of  their  weather  stations  on  the  airfield.  This  will  require  you  complete 
an  analysis  using  AOMS.  Your  customer  does  not  know  any  identifying  information  for 
the  weather  station  so  you  will  have  to  use  the  mapping  capability  to  locate  the  weather 
station. 

1 .  If  the  ArcMap  view  is  not  open,  then  Open  the  GIS  feature  from  AOMS  Launcher 
Menu. 

2.  You  determine  that  the  easiest  way  to  find  the  weather  stations  is  to  use  the 
Browse  Obstructions  feature.  Select  the  Browse  Obstructions  button  from  the 
toolbar.  This  will  assist  in  finding  the  correct  weather  station.  You  can  move  the 
Browse  Obstructions  window  to  another  location  on  the  existing  window. 

3.  You  select  the  first  Weather  Station  listed  in  the  list  of  obstructions.  To  more 
easily  identify  the  obstruction,  select  the  Highlight  Selected  Features  with  Arrows 
button  (yellow  arrow)  on  the  toolbar.  Your  customer  tells  you  that  this  is  not  the 
Weather  Station  she  is  inquiring  about. 

4.  You  select  the  next  Weather  Station  listed  in  the  list  of  obstructions.  The  number 
is  NC-18. 

5.  Select  the  Media  Window  button  (looks  like  a  camera)  from  the  toolbar.  This 
opens  a  picture  of  the  obstruction.  You  can  click  inside  the  picture  to  get  a  larger 
view  (if  you  do  this  you  will  have  to  close  the  window  to  see  the  obstruction 
location  on  the  ArcMap  view. 

6.  You  can  also  identify  this  obstruction  with  the  yellow  arrow.  Your  customer  tells 
you  that  this  is  the  correct  Weather  Station. 

7.  Your  customer  wants  the  distance  information  and  violation  information  of  the 
Weather  Station.  Using  the  Zoom  buttons  on  the  toolbar,  zoom  in  on  the 
obstruction. 
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8.  Find  and  select  the  Analyze  Location  button  from  the  toolbar.  This  opens  the 
AOMS  analysis  window  and  highlights  the  centerline  of  the  main  runway.  The 
analysis  information  can  be  determined  from  any  runway  that  is  loaded. 

9.  Select  the  obstruction  using  the  crosshair  selection  tool.  The  obstruction 
information  is  now  updated  in  the  AOMS  Analysis  window.  You  can  now  scroll 
through  the  tabs  on  the  AOMS  Analysis  window  and  provide  the  customer  with 
the  necessary  infonnation.  You  can  now  close  any  of  the  opened  AOMS  features. 


These  scenarios  have  been  designed  to  give  you  some  hands-on  interaction  with  the 
application.  You  should  now  be  able  to  complete  the  application  evaluation 
questionnaire.  If  you  need  to  use  the  application  to  assist  in  completing  the 
questionnaire,  please  do  so.  Your  thorough  investigation  of  the  application  is  critical  to 
providing  an  appropriate  assessment.  When  completed,  please  close  all  programs  and 
return  the  computer  to  the  Desktop  configuration. 
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Appendix  C:  Airfield  Obstruction  Tracking,  Analysis,  and  Management  System 

(AIROBS)  Scenarios 


These  scenarios  have  been  designed  to  give  you  some  hands-on  interaction  with  the 
application.  The  scenarios  will  give  you  some  familiarization  with  a  few  of  the  most 
important  requirements  expected  of  an  airfield  obstruction  management  system.  If  you 
feel  it  is  necessary,  please  take  notes  throughout  the  completion  of  the  scenarios.  Upon 
completion  of  the  scenarios  you  should  be  able  to  complete  the  evaluation  questionnaire. 
You  may  also  review  the  questionnaire  prior  to  or  during  the  completion  of  the  scenarios. 
If  you  think  that  more  interaction  is  needed,  feel  free  to  spend  additional  time  with  the 
application. 

Log  into  the  computer  using  the  following  information: 

User  N  ame :  Tester 

Password:  XSW@cde3 

Log  on  to:  CEXP32XY  (this  computer) 

Scenario  1 


You’ve  recently  arrived  at  your  new  duty  station,  XYZ  AFB.  You  are  assigned  to  the 
Engineering  Flight  and  the  Flight  Chief  wants  you  to  have  some  familiarity  with  all  of  the 
areas  within  the  flight.  You  are  going  to  be  working  with  the  community  planner  to  get 
some  familiarization  with  that  position.  One  of  the  responsibilities  of  the  community 
planner  is  to  track,  manage,  and  analyze  airfield  obstructions.  There  is  a  fairly  new 
system  in  place  which  assists  in  accomplishing  this  task.  AIROBS  is  the  program  used  at 
XYZ  AFB.  The  application  uses  Microsoft  Access  as  its  database  and  ESRI  ArcView  8.3 
for  its  mapping  capability. 

7.  Open  AIROBS  from  the  desktop  (this  will  take  approximately  one  minute). 

8.  Select  HELP  on  the  AIROBS  menu. 

9.  Select  the  AIROBS  User  Manual  and  spend  a  brief  amount  of  time  in  the 
following  areas: 

o  How  do  I  enter  a  new  obstruction?  (pg  10) 
o  How  do  I  modify  the  information  about  an  obstruction?  (pg  14) 
o  How  do  I  delete  an  obstruction?  (pg  15) 
o  How  can  I  create  a  Form  583?  (pg  17) 
o  Edit  existing  obstructions  (pg  47) 
o  Query  obstructions  (pg  5 1) 
o  Reports  (pg  52) 

10.  On  the  ArcMap  view,  pan  over  the  buttons  in  order  to  get  the  descriptions  of  each 
button’s  function. 

11.  When  finished  in  the  AIROBS  User  Manual,  close  the  Acrobat  Reader. 
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You  should  be  back  at  the  AIROBS  main  menu  area. 


Scenario  2 


The  Flight  Chief  comes  over  and  wants  you  to  load  a  new  obstruction  that  the  airfield 
manager  has  reported  to  him. 

8.  Select  Tools,  and  then  select  Analyze  Obstructions  on  the  AIROBS  menu. 

9.  In  the  Analyze  Obstructions  box,  fill  in  the  following  information. 

a.  Select  New  Siting  Analysis 

b.  For  the  purpose  of  this  exercise,  select  any  area  on  the  map  outside  of  the 
Analyzed  Obstructions  box.  Once  a  site  is  selected  the  X  Coord:,  Y 
Coord:,  and  Ground  Elevations  should  auto  fill.  The  height  of  the 
obstruction  must  be  filled  in  to  run  the  analysis. 

c.  Height:  3.0 

d.  UFC  needs  to  be  moved  from  Applicable  Criteria  to  Selected  for  Analysis 

e.  In  the  Selected  Analysis  area,  check  the  boxes  for  Airspace  Surface  and 
Taxiway/Apron  clearance. 

f.  Select  the  Analyze  button. 

g.  When  the  Result  Summary  appears,  select  Save  Data  to  DB.  At  this  time 
the  Add  New  Structure  window  opens.  Fill  in  the  following  information: 
Obstruction  Number:  Use  any  number  between  1-100 
Obstruction  Classification:  Temporary  Waiver 

Waiver  Status:  Pending 
Date:  Today’s  Date 
Construction  Type:  Permanent 

Obstruction  Description  and  Comments:  metal  pole  sticking  out  of 
ground 

Frangible:  No 

h.  Select  the  OK  button 

The  Add  New  Structure  window  closes  and  the  Analyze  Obstructions  window 
appears.  Close  this  window. 

This  is  all  that  is  needed  to  load  the  obstruction  data.  The  Engineering  Assistant  will 
collect  and  complete  the  identifying  information  for  the  obstruction. 

Scenario  3 


It  is  now  at  the  end  of  the  day  and  you  receive  a  call  from  the  Horizontal  shop  foreman. 
He  informs  you  that  the  metal  pole  sticking  out  of  the  ground  on  the  airfield  has  been 
removed.  Since  the  obstruction  was  taken  care  of  so  quickly,  it  is  not  necessary  to 
maintain  the  information  in  AIROBS. 

8.  From  the  AIROBS  main  menu  select  Edit  Obstructions. 
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9.  Scroll  through  the  list  and  find  the  obstruction  that  you  previously  loaded. 

10.  Select  the  obstruction  then  select  the  Delete  button. 

1 1 .  A  Windows  dialog  box  will  appear  and  ask  “Are  you  sure  you  want  to  delete  the 

obstruction  from  the  database?” 

12.  Select  Yes,  then  close  the  Edit  Obstructions  window. 

Scenario  4 


The  Airfield  Manager  has  requested  a  summary  report  of  all  obstructions.  The  Summary 
Report  can  be  generated  from  the  AIROBS  main  menu  or  the  ArcMap  Tools  drop  down. 

10.  Select  Forms  and  Reports  from  the  AIROBS  main  menu  or  select  Reports  from 
the  Tools  drop  down  list  on  the  ArcMap  view. 

11.  For  the  Report  Type:  Select  583  (Annual  Waiver  Report  Summary) 

12.  For  Save  Path:  You  can  leave  it  at  the  default  or  change  where  the  report  will  be 
saved.  Remember  where  it  is  saved. 

13.  Select  Next,  then  for  Select  Structures,  choose  All 

14.  Select  Finish 

15.  Once  the  report  is  finished  you  should  receive  a  Report  Successfully  Created 
dialog  box. 

16.  Close  the  Fonns  and  Reports  window. 

17.  You  can  now  go  to  the  report  where  it  was  saved  and  view  the  583  Summary 
Report. 

Scenario  5 


An  individual  from  the  Weather  Squadron  is  in  your  office  requesting  some  information 
concerning  a  weather  antenna  on  the  airfield.  This  will  require  you  complete  an  analysis 
using  AIROBS.  Your  customer  does  not  know  any  identifying  information  for  the 
weather  antenna  so  you  will  have  to  use  the  mapping  capability  to  locate  the  weather 
antenna. 

1 .  You  detennine  that  the  easiest  way  to  find  the  weather  antenna  is  to  use  the  Edit 
Obstructions  feature.  From  the  AIROBS  main  menu,  select  Tools  and  then  the 
Edit  Obstructions  feature.  This  will  assist  in  finding  the  correct  weather  station. 

2.  Scroll  through  the  list  of  obstructions  until  you  find  the  first  Weather  Antenna.  It 
should  be  obstruction  number  ACAD01025.  You  select  the  first  Weather 
Antenna  listed  in  the  list  of  obstructions. 

3.  Once  the  Weather  Antenna  is  selected  it  is  identified  on  the  ArcMap  view  with  a 
light  blue  arrow.  You  will  have  to  close  the  Edit  Obstructions  window  in  order  to 
move  the  AIROBS  main  menu  window  out  of  the  way. 

4.  When  the  windows  have  been  moved,  you  can  use  the  Zoom  Out  feature  (zoom 
out  about  3  times)  to  locate  the  light  blue  arrow  identifying  the  obstruction. 
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5.  When  the  light  blue  arrow  is  located,  your  customer  informs  you  that  the  weather 
antenna  is  the  correct  one.  Now  you  will  need  to  get  the  obstruction  information 
for  your  customer. 

6.  Find  the  Identify  Obstruction  button  on  the  ArcMap  toolbar  (fourth  button  from 
the  right). 

7.  After  the  Identify  Obstruction  button  is  selected,  (using  the  mouse)  place  the 
arrow  on  the  obstruction  identified  by  the  blue  arrow  and  select  the  obstruction. 

8.  The  Edit  Obstruction  window  opens  with  all  of  the  identifying  infonnation  for  the 
obstruction. 

9.  Based  on  the  ArcMap  location,  the  correct  Weather  Antenna  has  been  located. 
Your  customer  takes  all  the  needed  information  for  the  Weather  Antenna  and 
thanks  you  for  your  help. 

These  scenarios  have  been  designed  to  give  you  some  hands-on  interaction  with  the 
application.  You  should  now  be  able  to  complete  the  application  evaluation 
questionnaire.  If  you  need  to  use  the  application  to  assist  in  completing  the 
questionnaire,  please  do  so.  Your  thorough  investigation  of  the  application  is  critical  to 
providing  an  appropriate  assessment.  When  completed,  please  close  all  programs  and 
return  the  computer  to  the  Desktop  configuration. 
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Appendix  D:  Airfield  Obstruction  Management  Systems  Evaluation  Questionnaire 


Purpose:  To  conduct  a  comparative  analysis  of  two  Geographic  Information  System 
(GIS)  applications;  the  Airfield  Obstruction  Management  System  (AOMS)  and  the 
Airfield  Obstruction  Tracking,  Analysis,  and  Management  System  (AIROBS) 

Participation:  I  appreciate  your  participation  in  this  research  effort.  Your  participation 
is  completely  voluntary. 

Confidentiality:  I  ask  for  some  demographic  information  in  order  to  interpret  the  results 
more  accurately.  All  answers  are  anonymous. 

Contact  Information:  If  you  have  any  questions  or  comments  about  the  evaluation, 
please  contact  me. 


SMSgt  Mark  E.  Barner 
AFIT/ENV 
2950  Hobson  Way 
WPAFB,  OH  45433-7765 
Email:  mark.bamer@afit.edu 


INSTRUCTIONS 

•  Base  your  answers  on  your  experience  with  the  application 

•  Please  print  your  answers  clearly  when  asked  to  write  a  response  or  when 
providing  comments 

•  Make  dark  marks  when  asked  to  use  specific  response  options 

•  Avoid  stray  marks.  If  you  make  corrections,  erase  marks  completely  or 
clearly  indicate  the  intended  response  if  you  use  an  ink  pen 

MARKING  EXAMPLES 


SOFTWARE  USABILITY  EVALUATION 
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INSTRUCTIONS  TO  RESPONDENTS 


This  questionnaire  contains  a  series  of  questions  about  the  usability  of  the  software  you 
will  be  testing.  In  general,  “usability”  simply  means  how  easy  the  software  is  to  use.  In 
this  questionnaire,  usability  will  be  assessed  in  terms  of  presence  or  absence  of  a  set  of 
software  quality  features  that  are  indicative  of  good  interface  design.  For  each 
assessment  item,  you  will  be  asked  to  indicate  whether  or  not  the  listed  design  feature  is 
consistently  implemented  in  the  software  interface  being  evaluated. 

Responses  are  to  be  made  by  darkening  the  circle  below  or  adjacent  to  the  descriptor  that 
best  expresses  your  opinion.  The  initial  response  is  to  indicate  whether  the  design  feature 
is  either  Always  or  Not  Always  present  in  the  software  interface  you  are  evaluating.  The 
additional  choices  of  Don  ’t  Know  and  Not  Applicable  may  also  be  selected,  if 
appropriate. 

Where  a  design  feature  is  lacking  (i.e.,  your  response  is  Not  Always),  please  provide  an 
estimate  of  the  difficulty  caused  by  the  absence  of  that  feature.  These  ratings  are 
expressed  in  terms  of  the  impact  on  operational  effectiveness,  ranging  from  Very  Low  to 
Very  High.  Taking  notes  while  working  through  the  scenarios  may  assist  in  completing 
the  questionnaire. 

Where  a  design  feature  is  lacking,  you  are  also  asked  to  provide  an  example  of  where  or 
how  that  feature  is  lacking.  Such  an  example  might  be:  “The  cursor  is  difficult  to  locate 
on  the  screen.” 

In  addition  to  the  usability  of  the  application,  questions  36-43  address  End-User 
Computing  Satisfaction.  These  items  use  a  Likert  scale  ranging  from  (1)  Almost  Never 
to  (5)  Almost  Always. 

The  estimated  time  to  complete  the  evaluation  is  1  hour  30  minutes. 

Please  answer  all  the  questions  at  your  own  pace.  If  you  have  any  uncertainty  about  the 
meaning  of  terms  or  the  intent  of  the  questions,  please  contact  the  questionnaire 
administrator, 

SMSgt  Mark  Barner. 


Thank  you  for  your  assistance! 


PROCEED  TO  THE  NEXT  PAGE 


65 


1 .  The  user  is  adequately  prompted  as  to  where  on  the  display  data  are  to  be  entered. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 

o  o  o  o  o 

2.  Input  devices  (e.g.,  keyboard,  cursor,  mouse)  are  appropriate  for  the  tasks  being 


performed. 

ALWAYS 

NOT  ALWAYS 

DON’T  KNOW 

NOT  APPLICABLE 

O 

O 

O 

O 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


3.  The  cursor  is  easy  to  locate. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 
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4.  Data  fields  are  adequately  labeled. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 

o  o  o  o  o 

5.  Text  inputs  are  easy  to  edit. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH 

o  o  o  o 


VERY  HIGH 

O 


6.  Where  data  are  entered  from  source  documents,  the  format  for  data  entry  corresponds 
to  that  of  the  source  documents. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 
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7.  The  system  provides  quick,  positive  feedback  on  the  acceptance  or  rejection  of  data 
entry. 


ALWAYS 

NOT  ALWAYS 

DON’T  KNOW 

NOT  APPLICABLE 

O 

O 

O 

O 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


8.  The  system  validates  user  inputs  before  processing  them. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 

o  o  o  o  o 

9.  The  system  provides  adequate  notification  when  it  detects  a  data  entry  error. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 
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10.  Menu  selection  options  are  readily  understandable. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 

o  o  o  o  o 

11.  Menu  selection  options  are  logically  organized  by  similarity  of  function  and/or  by 


order  of  use. 

ALWAYS 

NOT  ALWAYS 

DON’T  KNOW 

NOT  APPLICABLE 

O 

O 

O 

O 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


12.  Wording  of  menu  options  is  consistent  with  the  functions  and  processes  they  control. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 
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13.  The  system  responds  quickly  and  accurately  to  menu  commands. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


14.  The  system  responds  quickly  and  accurately  to  mouse  selection  button  presses. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


15.  Manipulation  of  objects  does  not  require  excessively  fine  pointing  or  manual 


adjustment. 

ALWAYS 

NOT  ALWAYS 

DON’T  KNOW 

NOT  APPLICABLE 

O 

O 

O 

O 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 
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16.  The  status  of  manipulated  objects  (e.g.,  active,  selected,  unavailable)  is  clearly 


displayed. 

ALWAYS 

NOT  ALWAYS 

DON’T  KNOW 

NOT  APPLICABLE 

O 

O 

O 

O 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


17.  Information  is  easy  to  find  on  the  screen. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


18.  The  amount  of  data  presented  at  any  one  time  is  appropriate. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 
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19.  Data  entry  errors  are  easy  to  correct. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


20.  Data  display  fonnats  are  consistent  across  the  system. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


21.  Wording  is  consistent  across  displays. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 


VERY  LOW  LOW 


MEDIUM  HIGH 


O 


O 


o 


o 


VERY  HIGH 

O 
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22.  Text  displays  are  easy  to  read. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 

o  o  o  o  o 

23.  The  method  for  controlling  windows  is  consistent  across  displays. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


24.  Window  overlays  are  situated  so  that  they  do  not  obscure  important  information. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 


VERY  LOW  LOW 


MEDIUM  HIGH 


O 


O 


o 


o 


VERY  HIGH 

O 
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25.  Where  several  windows  are  displayed  simultaneously,  it  is  easy  for  the  user  to  shift 
among  them  to  select  which  window  is  to  be  made  active. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 

o  o  o  o  o 

26.  Formats  for  data  entry  are  consistent  across  different  displays. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


27.  Schematic  and  pictorial  displays  are  clearly  drawn  and  labeled. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 
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28.  Graphic  symbology  is  appropriate  for  the  information  it  represents. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


29.  Colors  used  in  displays  are  easy  to  distinguish  from  one  another. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


30.  Color  coding  is  used  consistently  across  different  displays. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 
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3 1 .  The  system  provides  adequate  feedback  when  an  internal  fault  is  detected. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


32.  The  system  provides  an  adequate  amount  of  on-line  user  guidance. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


33.  On-line  user  guidance  is  readily  understandable. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 
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34.  System  functions  are  organized  in  a  manner  that  is  consistent  with  the  tasks  they  are 
designed  to  perform. 

ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


35.  The  system  provides  the  user  with  all  of  the  information  needed  to  perform  required 
tasks. 


ALWAYS  NOT  ALWAYS  DON’T  KNOW  NOT  APPLICABLE 

o  o  o  o 

For  a  response  of  “NOT  ALWAYS”,  please  provide  an  example  of  where  the  feature  is  lacking: 


What  is  the  overall  impact  of  this  feature  on  operational  effectiveness? 

VERY  LOW  LOW  MEDIUM  HIGH  VERY  HIGH 


O 


O 


o 


o 


o 


PLEASE  PROCEED  TO  THE  NEXT  PAGE 
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For  each  statement  below;  please  fill  in  the  circle  for  the  number  that  indicates  the 
extent  to  which  you  agree  with  each  statement.  Use  the  scale  below  for  your 

responses. 


<£>  ©  ®  © 

..  Some  of  the  Time  About  Half  ,  r,_ 

Almost  Never  Most  of  the  Time 

Ol  the  Time 

(D 

Almost  Always 

36.  Does  the  application  provide  the  precise  information  you  need? 

®  ©  (D  ©  © 

37.  Does  the  information  content  meet  your  needs? 

®  ©  (D  ©  (D 

38.  Does  the  application  provide  reports  that  seem  to  be  just  about 
exactly  what  you  need? 

®  ®  (3)  ©  © 

39.  Does  the  application  provide  sufficient  information? 

®  ®  (D  ©  © 

40.  Do  you  think  the  output  is  presented  in  a  useful  manner? 

®  ©  (D  ©  (D 

41.  Is  the  information  clear? 

®  ©  (D  ©  (D 

42.  Is  the  application  user  friendly? 

®  ©  (D  ©  © 

43.  Is  the  application  easy  to  use? 

®  ®  (D  ©  © 

This  section  contains  items  regarding  personal  work  related  information.  Respond  to 
each  item  by  WRITING  in  the  information  requested  or  FILLING  in  the  corresponding 
circles  that  best  describe  you. 

44.  What  is  your  current  rank? 


0-1 

0-2 

0-3 

0-4 

o 

o 

o 

o 

45.  What  is  your  total  time-in-service  (Total  Federal  Active  Military  Service)? 

Years _  Months _ 

46.  Which  Civil  Engineer  Flights  have  you  been  previously  assigned  (mark  all  that  apply)? 


EOD 

Engineering 

Environmental 

Fire  Department 

O 

O 

O 

O 

Housing 

Operations 

Readiness 

Resources 

O 

O 

O 

O 
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PLEASE  PROCEED  TO  THE  NEXT  PAGE 


47.  Have  you  worked  with  Geographic  Information  Systems/GeoBase  applications 
previously? 

Yes  No 

O  O 


48.  How  would  you  rate  your  level  of  experience  with  GIS/GeoBase  applications? 

None  Some  Moderate  Extensive 

o  o  o  o 


Questions/Concerns 

If  you  have  any  questions  or  concerns  please  feel  free  to  contact  me.  My  contact 
information  is  on  the  cover  sheet.  I  appreciate  your  participation  and  would  be  happy  to 
address  any  questions  you  may  have  regarding  the  questionnaire  or  the  research  in 
general. 


Thanks  again  for  your  assistance  in  this  research  project. 


Please  leave  completed  questionnaire  in  the  evaluation  room  or 
return  completed  questionnaire  to  SMSgt  Barner. 
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Appendix  E:  E-mail  Request  for  Evaluator  Support 


To  All, 

I  am  a  fellow  CE  troop  (enlisted)  working  on  my  AFIT  thesis  project.  The  project 
involves  evaluating  two  airfield  obstruction  management  systems  currently  in  use  by  CE 
Community  Planners  across  the  Air  Force.  I  know  everyone  is  very  busy  with  classes  in 
the  CESS  or  with  preparing  and  taking  final  exams,  but  I  would  sincerely  appreciate  your 
help  in  conducting  these  evaluations. 

This  research  was  requested  to  be  conducted  by  the  Headquarters  Air  Force  Geo 
Integration  Office  (HAF  GIO)  and  is  also  supported  by  the  GIS  Support  Center.  Benefits 
of  this  research  may  include;  identifying  which  application  is  superior  in  usability; 
identifying  a  standard  application  for  AF  wide  use;  and  providing  a  methodology  to 
assess  other  applications.  As  a  CE  officer  this  evaluation  would  also  give  you 
familiarization  to  a  program  that  you  could  be  responsible  for  in  the  future. 

It  is  not  required  that  you  have  previous  knowledge  of  airfield  obstruction  management 
systems.  Your  task  will  be  to  complete  the  scenarios  and  evaluation  questionnaire  of  one 
of  the  applications.  The  scenarios  have  been  designed  to  give  you  some  hands-on 
interaction  with  the  application.  The  scenarios  will  give  you  some  familiarization  with  a 
few  of  the  most  important  requirements  expected  of  an  airfield  obstruction  management 
system.  Upon  completion  of  the  scenarios,  you  should  have  no  problem  completing  the 
evaluation  questionnaire.  The  interaction  and  questionnaire  completion  should  take 
approximately  1  hour  and  30  minutes.  I  realize  this  is  a  lot  of  time  to  ask  of  you,  but  it  is 
necessary  in  order  for  an  appropriate  assessment  to  be  accomplished. 

The  applications  are  loaded  on  two  computers  located  in  the  small  conference  room,  third 
floor  of  the  CESS.  Available  dates/times  to  complete  the  evaluations  are: 

Friday,  9  Dec  1000  -  1600  hrs 

Monday,  12  Dec  -  Friday,  16  Dec  0800-1600  hrs 

Please  consider  my  request  for  support  with  this  project.  If  you  would  like  to  assist 
please  reply  to  this  email  and  provide  a  block  of  time  that  is  convenient  for  you. 

SMSgt  Mark  E.  Barner 
AFIT/ENV 
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Appendix  F:  AOMS  Not  Always  Responses 


AOMS-Usability  Statements  with  Not  Always  Responses 

Item 

Attribute 

Function 

Statement!  Comments 

Impact 

1 

Descriptiveness 

Data  Entry 

The  user  is  adequately  prompted  as  to  where  on  the 
display  data  are  to  be  entered. 

a.  New  Obstruction  screen  could  have  more 
guidance/instruction. 

High 

b.  Field  definitions  are  not  always  clear  thus  making 
it  difficult  to  know  what  needs  to  be  filled  and  when. 

High 

4 

Descriptiveness 

Data  Entry 

Data  fields  are  adequately  labeled. 

Field  definitions  are  not  always  clear  thus  making  it 
difficult  to  know  what  needs  to  be  filled  and  when. 

High 

35 

Descriptiveness 

Mission 

Performance 

The  system  provides  the  user  with  all  of  the 
information  needed  to  perform  required  tasks. 

a.  Field  sometimes  cryptic  but  drop-down  menus 
helpful. 

High 

b.  I  could  not  have  done  the  tasks  if  they  were  not  in 
a  step  by  step  format. 

High 

24 

Descriptiveness 

Data 

Display 

Window  overlays  are  situated  so  that  they  do  not 
obscure  important  information. 

a.  Menus  can  be  move  which  is  ok. 

High 

b.  Pictures  of  obstructions  popped  up  in  middle  of 
screen 

Low 

c.  Window  displaying  obstruction  text  data  may 
obscure  location  on  map.  This  easily  corrected  by 
ease  of  moving  window. 

Medium 

d.  Browse  Obstruction  &  Media  widow  blocked  map 
data. 

Very 

Low 

e.  The  ‘browse  obstruction’  and  ‘analyze  location’ 
menus  cover  up  map.  Maybe  a  side  bar  could  pop 
up  that  could  hold  these  menus. 

High 
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27 

Descriptiveness 

Data 

Display 

Schematic  and  pictorial  displays  are  clearly  drawn 
and  labeled. 

a.  Display  example  difficult  to  follow. 

Medium 

b.  The  map  did  not  have  any  labels;  that  could  be 
helpful. 

Medium 

10 

Descriptiveness 

Interactive 

Control 

Menu  selection  options  are  readily  understandable. 

Very 

Low 

a.  Media  window  will  not  show  picture  if  highlight 
feature  w/arrow  is  on. 

b.  Didn’t  understand  difference  between  “Browse 
Obstructions”  &  “Search  Obstructions.” 

Very 

Low 

16 

Descriptiveness 

Interactive 

Control 

The  status  of  manipulated  objects  (e.g.,  active, 
selected,  unavailable)  is  clearly  displayed. 

In  ArcMap’s  Analyze  Location  window  the  status  of 
objects  is  shown  to  be  unavailable. 

Low 

5 

Responsiveness 

Data  Entry 

Text  inputs  are  easy  to  edit. 

a.  You  might  want  to  have  a  spell  check  option  it 
that  is  important. 

High 

b.  Can’t  edit  inside  Browse  Obstructions  window, 
need  to  open  up  the  obstruction’s  report  first. 

Very 

Low 

13 

Responsiveness 

Interactive 

Control 

The  system  responds  quickly  and  accurately  to  menu 
commands. 

a.  GIS  button  in  AOMS  Launcher  was  slow. 

High 

b.  Only  exception  was  initial  entry  into  the  system 
which  took  a  long  time. 

Medium 

c.  Slow  to  open  GIS  from  Launcher  menu. 

Very 

High 

d.  GIS  took  few  minutes  to  load,  system  dependent. 

Medium 
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14 

Responsiveness 

Interactive 

Control 

The  system  responds  quickly  and  accurately  to 
mouse  selection  button  presses. 

The  photo  of  the  obstruction  didn’t  load  the  first 
time  and  had  to  be  reloaded. 

Medium 

15 

Responsiveness 

Interactive 

Control 

Manipulation  of  objects  does  not  require  excessively 
fine  pointing  or  manual  adjustment. 

Found  it  difficult  to  do  graphic  walk  through. 

High 

23 

Consistency 

Data 

Display 

The  method  for  controlling  windows  is  consistent 
across  displays. 

After  closing  the  Analyze  Location  window,  I  was 
not  able  to  reopen  it.  It  appears  the  system  still 
thinks  its  open. 

Very 

High 

30 

Consistency 

Data 

Display 

Color  coding  is  used  consistently  across  different 
displays. 

What  did  the  colors  on  the  map  mean? 

Medium 

34 

Consistency 

Mission 

Performance 

System  functions  are  organized  in  a  manner  that  is 
consistent  with  the  tasks  they  are  designed  to 
perform. 

See  1  la.  and  17a. 

Very 

Low 

19 

Error 

Abatement 

Data  Entry 

Data  entry  errors  are  easy  to  correct. 

Easy  to  get  lost  in  the  layers. 

High 

29 

Simplicity 

Data 

Display 

Colors  used  in  displays  are  easy  to  distinguish  from 
one  another. 
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a.  Light  blue  dot  representing  obstruction  on  may 
was  difficult  to  spot.  Compensated  by  large  yellow 
arrow  pointing  to  obstruction  when  this  tool  was 
selected. 

Medium 

b.  Map  was  pastel  and  very  hard  to  see  what  was 
going  on.  When  you  scroll  up  or  down.  New  area  is 
a  lighter  color.  Very  hard  to  make  heads  or  tails  of. 

High 

17 

Simplicity 

Data 

Display 

Information  is  easy  to  find  on  the  screen. 

a.  Drop  down  menus  are  not  explanatory  or  easy. 

Tabs  were  good  most  of  the  time,  but  Analyze 
Location  information  was  a  little  confusing. 

Very 

Low 

b.  Not  the  first  time  around,  but  gets  easier 
w/exposure. 

Very 

Low 

c.  The  Analyze  Location  function  is  hard  to 
understand. 

High 

11 

Simplicity 

Interactive 

Control 

Menu  selection  options  are  logically  organized  by 
similarity  of function  and/or  by  order  of  use. 

a.  Some  of  the  buttons  (i.e.,  highlight  feature  with 
arrow)  should  be  next  to  Browse  Obstruction  button. 
Also,  could  not  find  Analyze  location  on  drop  down 
menu). 

Medium 

b.  I  had  expected  the  Help  Menu  to  be  at  the  top  of 
the  screen  as  in  MS  applications. 

Very 

Low 

c.  Zoom  is  not  on  the  edit  toolbar. 

Medium 

25 

Simplicity 

Data 

Display 

Where  several  windows  are  displayed 
simultaneously,  it  is  easy  for  the  user  to  shift  among 
them  to  select  which  window  is  to  be  made  active. 

Yes,  except  main  ArcMap  buttons  cannot  be 
highlighted  to  see  what  they  are  when  using  extra 
windows  (i.e.  browse  obstruction),  but  does  not 
show  on  bottom  of  screen  so  I  guess  it’s  ok. 

Medium 
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Appendix  G:  AIROBS  Not  Always  Responses 


AIROBS-Usability  Statements  with  Not  Always  Responses 

Item 

Attribute 

Function 

Statement!  Comments 

Impact 

16 

Descriptiveness 

Interactive 

Control 

The  status  of  manipulated  objects  (e.g.,  active, 
selected,  unavailable)  is  clearly  displayed. 

I  could  not  immediately  see  the  light  blue  arrow. 

Had  to  really  look  for  it  (weather  antenna). 

Very  High 

27 

Descriptiveness 

Data 

Display 

Schematic  and  pictorial  displays  are  clearly  drawn 
and  labeled. 

a.  The  obstructions  did  not  seem  to  be  labeled  on 
the  map. 

High 

b.  Difficult  to  see  light  blue  arrow. 

Very  High 

24 

Descriptiveness 

Data 

Display 

Window  overlays  are  situated  so  that  they  do  not 
obscure  important  information. 

a.  When  selecting  an  airfield  obstruction  and  then 
trying  to  locate  the  obstruction  on  the  map,  you  had 
to  move  the  “Edit  Obstruction”  and  “Main  Menu” 
dialogue  box. 

Medium 

b.  I  had  to  manually  move  the  windows  to  look  at 
the  Arc  View  map. 

Medium 

c.  The  AIROBS  menu  can  be  much  smaller. 

Medium 

d.  “Edit  Obstructions”  window  had  to  be  closed  to 
select  weather  antenna  in  scenario  5.  Ability  to  just 
minimize  might  be  helpful. 

Medium 

e.  Need  to  close  screens  to  view  map. 

Medium 

f.  You  have  to  minimize  or  move  windows  to  see 
the  map. 

High 

g.  Should  be  able  to  move  sub  windows  to  view 
map.  I  had  to  close  multiple  windows  to  view  map. 
Many  software  programs  allow  you  to  minimize 
worksheets  that  would  be  helpful/convenient. 

High 

h.  I  had  a  lot  of  difficulty  with  viewing  map  & 
getting  rid  of  overlays. 

Medium 
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i.  Had  to  close  dialog  boxes  to  see  selected  item. 
Would  rather  “minimize”  and  use  ArcView. 

Very  High 

4 

Descriptiveness 

Data  Entry 

Data  fields  are  adequately  labeled. 

In  some  cases  the  fields  are  labeled  using  acronyms 
which  may  be  confusing  for  a  new  airfield  manager 
(DB,  FIM,  etc.) 

High 

1 

Descriptiveness 

Data  Entry 

The  user  is  adequately  prompted  as  to  where  on  the 
display  data  are  to  be  entered. 

a.  When  you  type  “Temporary  Waiver”  instead  of 
using  the  drop  down  menu,  the  program  doesn’t 
recognize  the  entry. 

High 

b.  Add  new  structure  -  what  is  “obstruction 
frangible”  used  for?  Prompt  occurs  only  when 
infonnation  is  missing. 

Medium 

c.  Confusion  with  “Obstruction  Frangible”  and 
“Frangible”  field.  Thought  both  were  the  same 
thing. 

Low 

31 

Descriptiveness 

User 

Guidance 

The  system  provides  adequate  feedback  when  an 
internal  fault  is  detected. 

I  could  not  zoom  in  or  out  and  there  was  an  error. 
However,  no  error  message  appeared. 

High 

35 

Descriptiveness 

Mission 

Performance 

The  system  provides  the  user  with  all  of  the 
information  needed  to  perform  required  tasks. 

a.  Sometimes  I  didn’t  know  exactly  what  it  was 
talking  about  in  the  non-critical  selections. 

Medium 

b.  I  don’t  think  some  of  the  functions  I  performed 
(adding  a  new  obstruction,  finding  an  obstruction) 
were  prompted  well  by  the  system. 

Medium 

5 

Responsiveness 

Data  Entry 

Text  inputs  are  easy  to  edit. 

a.  The  directions  said  to  enter  “No”  in  the  Frangible 
box  and  this  box  would  not  accept  any  text. 

Low 
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b.  Need  to  differentiate  fields  that  can  be  user 
modified  and  those  that  are  calculated  by  the 
program. 

Medium 

7 

Responsiveness 

Data  Entry 

The  system  provides  quick,  positive  feedback  on  the 
acceptance  or  rejection  of  data  entry. 

a.  Sometimes  slow. 

Medium 

b.  First  time  user  might  be  unfamiliar  w/data  entry 
format  (i.e.  date).  Date  format  is  not  followed 
menu  prompts  “insert  date”. 

High 

c.  Scenario  #5  (2-4)  blue  arrow  did  not  appear 
quickly  -  could  not  be  found  for  2-3  minutes. 

Medium 

13 

Responsiveness 

Interactive 

Control 

The  system  responds  quickly  and  accurately  to 
menu  commands. 

a.  Scrolling  map  or  analyzing  data. 

Low 

b.  As  with  all  databases,  it  would  be  nice  to  have 
instant  feedback  from  the  system. 

High 

c.  See  #7c 

Medium 

14 

Responsiveness 

Interactive 

Control 

The  system  responds  quickly  and  accurately  to 
mouse  selection  button  presses. 

When  zooming,  screen  refresh  was  choppy  and 
sporadic. 

Medium 

15 

Responsiveness 

Interactive 

Control 

Manipulation  of  objects  does  not  require 
excessively  fine  poin  ting  or  manual  adjustmen  t. 

a.  The  zoom  features  took  some  getting  used  to. 

Very  Low 

b.  Not  able  to  click  on  item  on  map  w/blue  arrow  - 
nothing  happened. 

Medium 

11 

Simplicity 

Interactive 

Control 

Menu  selection  options  are  logically  organized  by 
similarity  of function  and/or  by  order  of  use. 

a.  Not  until  I  get  used  to  it.  Following  your 
instructions  made  it  easy. 

Medium 
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b.  Could  not  see  a  pattern. 

High 

c.  I  would  not  intuitively  think  to  use  the  “analyze 
obstructions”  menu  to  add  a  new  obstruction  to  the 
data  base. 

High 

17 

Simplicity 

Data 

Display 

Information  is  easy  to  find  on  the  screen. 

a.  When  selecting  an  airfield  obstruction  from  the 
edit  list,  it  was  difficult  to  locate  the  obstruction  on 
the  map  (couldn’t  find  the  blue  arrow),  should 
automatically  zoon  in  on  the  obstruction  desired. 

High 

b.  Finding  the  selected  obstruction  on  the  map 
required  zooming  and  searching. 

Medium 

c.  Marker  on  map  is  too  cluttered  when  many 
objects  are  mapped.  Either  reduce  marker  size. 

High 

d.  All  except  blue  arrow  (change  to  vibrant  color, 
red,  black,  etc... and  big). 

Very  High 

18 

Simplicity 

Data 

Display 

The  amount  of  data  presented  at  any  one  time  is 
appropriate. 

Should  be  able  to  filter  what  objects  are  presented 
on  map  readily  on  map  grid. 

High 

22 

Simplicity 

Data 

Display 

Text  displays  are  easy  to  read. 

a.  The  “distance  measurement”  on  the  edit 
obstructions  menu  was  very  small  &  hard  to  read 
quickly. 

Low 

b.  It  would  be  nice  for  reports  to  just  pop  up,  if  I’m 
querying,  more  than  likely,  I  want  it. 

Medium 

25 

Simplicity 

Data 

Display 

Where  several  windows  are  displayed 
simultaneously,  it  is  easy  for  the  user  to  shift  among 
them  to  select  which  window  is  to  be  made  active. 

a.  Not  all  the  windows  are  “minimizable.”  Example: 
the  “Edit  Obstructions”  window  where  you  are 
selecting  an  existing  obstruction. 

Medium 

b.  See  #24e. 

Medium 
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c.  This  is  lacking  between  the  map  and  the  menu 
screen. 

Medium 

d.  Some  dialog  boxes  remained  on  top  until  it  was 
closed  by  the  user. 

Low 

e.  See  #24h 

Medium 

f.  See  #24i 

High 

28 

Simplicity 

Data 

Display 

Graphic  symbology  is  appropriate  for  the 
information  is  represents. 

a.  For  example,  one  of  the  icons  on  the  map  screen 
is  an  “N”.  That  icon  is  for  “Obstruction  Analysis”. 
That  doesn’t  make  sense. 

Low 

b.  We  need  a  legend;  perhaps. 

Medium 

c.  No  idea  what  the  symbols  indicate. 

Medium 

d.  Graphics  are  not  labeled  on  the  map. 

Medium 

e.  Change  blue  arrow. 

High 

29 

Simplicity 

Data 

Display 

Colors  used  in  displays  are  easy  to  distinguish  from 
one  another. 

a.  Map  when  several  items  are  selected,  the  entire 
screen  gets  lighter  &  has  little  contrast. 

Medium 

b.  Could  not  easily  see  blue  arrow. 

High 

33 

Simplicity 

User 

Guidance 

On-line  user  guidance  is  readily  understandable. 

The  screen  shots  in  the  user  manual  aren’t  readable- 
maybe  blow  those  up-they  are  quicker  to  understand 
than  reading  the  text. 

Medium 

6 

Consistency 

Data  Entry 

Where  data  are  entered  from  source  documents,  the 
format  for  data  entry  corresponds  to  that  of  the 
source  documents. 

The  data  given  was  not  in  the  same  fonnat. 

Example:  there  were  two  input  blanks  for 
“frangible.” 

High 
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12 

Consistency 

Interactive 

Control 

Wording  of  menu  options  is  consistent  with  the 
functions  and  processes  they  control. 

a.  See  #1  lc 

High 

b.  The  ArcMap  menu  toolbar  is  good  and  the 
combining  of  ArcMap  menu  options  w/AIROBS  is 
also  good.  If  differentiating  the  two  sets  of  icons 
from  each  other  is  possible,  it  may  make  the  toolbar 
more  intuitive.  For  example,  the  “Add  Data”  button 
is  to  add  GIS  information  to  the  map  and  not  to  add 
airfield  obstruction  data. 

Low 

23 

Consistency 

Data 

Display 

The  method  for  controlling  windows  is  consistent 
across  displays. 

Once  a  user  uses  the  system  a  few  times,  he/she  will 
develop  his  own  methods  for  controlling  the 
windows.  Recommend  “Autoclosing”  “Forms  & 
Reports”  dialog  box  when  report  generation  is 
complete. 

Low 

34 

Consistency 

Mission 

Performance 

System  functions  are  organized  in  a  manner  that  is 
consistent  with  the  tasks  they  are  designed  to 
perform. 

The  tools  list  could  be  better  organized. 

Low 

8 

Error 

Abatement 

Data  Entry 

The  system  validates  user  inputs  before  processing 
them. 

a.  Only  for  critical  paths,  I  guess  it  can’t  be  as  smart 
as  us. 

High 

b.  I  saw  no  validation  process. 

Medium 
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Appendix  H:  Always  Responses  Received  by  Item 


Always  Responses  Received  by  Item 

Item  # 

Attribute 

Item 

AOMS 

AIROBS 

1 

Descriptiveness 

The  user  is  adequately  prompted  as  to 
where  on  the  display  data  are  to  be  entered. 

7 

7 

2 

Responsiveness 

Input  devices  (e.g.,  keyboard,  cursor, 
mouse)  are  appropriate  for  the  tasks  being 
perfonned. 

10 

10 

3 

Simplicity 

The  cursor  is  easy  to  locate. 

10 

10 

4 

Descriptiveness 

Data  fields  are  adequately  labeled. 

8 

8 

5 

Responsiveness 

Text  inputs  are  easy  to  edit. 

8 

8 

6 

Consistency 

Where  data  are  entered  from  source 
documents,  the  format  for  data  entry 
corresponds  to  that  of  the  source 
documents. 

6 

2 

7 

Responsiveness 

The  system  provides  quick,  positive 
feedback  on  the  acceptance  or  rejection  of 
data  entry. 

4 

5 

8 

Error 

Abatement 

The  system  validates  user  inputs  before 
processing  them. 

4 

4 

9 

Error 

Abatement 

The  system  provides  adequate  notification 
when  it  detects  a  data  entry  error. 

3 

6 

10 

Descriptiveness 

Menu  selection  options  are  readily 
understandable. 

8 

10 

11 

Simplicity 

Menu  selection  options  are  logically 
organized  by  similarity  of  function  and/or 
by  order  of  use. 

6 

7 

12 

Consistency 

Wording  of  menu  options  is  consistent 
with  the  functions  and  processes  they 
control. 

10 

8 

13 

Responsiveness 

The  system  responds  quickly  and 
accurately  to  menu  commands. 

6 

7 

14 

Responsiveness 

The  system  responds  quickly  and 
accurately  to  mouse  selection  button 
presses. 

9 

9 

15 

Responsiveness 

Manipulation  of  objects  does  not  require 
excessively  fine  pointing  or  manual 
adjustment. 

8 

7 

16 

Descriptiveness 

The  status  of  manipulated  objects  (e.g., 
active,  selected,  unavailable)  is  clearly 

8 

6 
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displayed. 

17 

Simplicity 

Information  is  easy  to  find  on  the  screen. 

7 

5 

18 

Simplicity 

The  amount  of  data  presented  at  any  one 
time  is  appropriate. 

10 

9 

19 

Error 

Abatement 

Data  entry  errors  are  easy  to  correct. 

7 

7 

20 

Consistency 

Data  display  formats  are  consistent  across 
the  system. 

9 

10 

21 

Consistency 

Wording  is  consistent  across  displays. 

9 

9 

22 

Simplicity 

Text  displays  are  easy  to  read. 

10 

8 

23 

Consistency 

The  method  for  controlling  windows  is 
consistent  across  displays. 

7 

9 

24 

Descriptiveness 

Window  overlays  are  situated  so  that  they 
do  not  obscure  important  infonnation. 

4 

1 

25 

Simplicity 

Where  several  windows  are  displayed 
simultaneously,  it  is  easy  for  the  user  to 
shift  among  them  to  select  which  window 
is  to  be  made  active. 

9 

4 

26 

Consistency 

Formats  for  data  entry  are  consistent  across 
different  displays. 

10 

10 

27 

Descriptiveness 

Schematic  and  pictorial  displays  are 
clearly  drawn  and  labeled. 

8 

6 

28 

Simplicity 

Graphic  symbology  is  appropriate  for  the 
infonnation  it  represents. 

10 

5 

29 

Simplicity 

Colors  used  in  displays  are  easy  to 
distinguish  from  one  another. 

6 

8 

30 

Consistency 

Color  coding  is  used  consistently  across 
different  displays. 

7 

9 

31 

Descriptiveness 

The  system  provides  adequate  feedback 
when  an  internal  fault  is  detected. 

2 

4 

32 

Descriptiveness 

The  system  provides  an  adequate  amount 
of  on-line  user  guidance. 

9 

7 

33 

Simplicity 

On-line  user  guidance  is  readily 
understandable. 

8 

7 

34 

Consistency 

System  functions  are  organized  in  a 
manner  that  is  consistent  with  the  tasks 
they  are  designed  to  perform. 

9 

8 

35 

Descriptiveness 

The  system  provides  the  user  with  all  of 
the  information  needed  to  perform  required 
tasks. 

6 

7 
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